dsh-data-quality 0.1.2 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +32 -0
- package/README.es.md +8 -2
- package/README.hi.md +8 -2
- package/README.md +22 -9
- package/README.pt.md +8 -2
- package/README.zh.md +8 -2
- package/cordis.patch.yml +11 -0
- package/lib/index.js +1411 -145
- package/lib/types/config.d.ts +12 -0
- package/lib/types/config.d.ts.map +1 -1
- package/lib/types/config.js +35 -0
- package/lib/types/config.js.map +1 -1
- package/lib/types/contract.d.ts +83 -0
- package/lib/types/contract.d.ts.map +1 -0
- package/lib/types/contract.js +144 -0
- package/lib/types/contract.js.map +1 -0
- package/lib/types/dataset.d.ts +50 -3
- package/lib/types/dataset.d.ts.map +1 -1
- package/lib/types/dataset.js +76 -21
- package/lib/types/dataset.js.map +1 -1
- package/lib/types/events.d.ts +2 -0
- package/lib/types/events.d.ts.map +1 -1
- package/lib/types/events.js.map +1 -1
- package/lib/types/index.d.ts +11 -8
- package/lib/types/index.d.ts.map +1 -1
- package/lib/types/index.js +15 -6
- package/lib/types/index.js.map +1 -1
- package/lib/types/presets.d.ts +28 -0
- package/lib/types/presets.d.ts.map +1 -0
- package/lib/types/presets.js +134 -0
- package/lib/types/presets.js.map +1 -0
- package/lib/types/profile.d.ts +38 -1
- package/lib/types/profile.d.ts.map +1 -1
- package/lib/types/profile.js +64 -10
- package/lib/types/profile.js.map +1 -1
- package/lib/types/provider-local.d.ts +7 -1
- package/lib/types/provider-local.d.ts.map +1 -1
- package/lib/types/provider-local.js +78 -4
- package/lib/types/provider-local.js.map +1 -1
- package/lib/types/scorecard.d.ts +50 -0
- package/lib/types/scorecard.d.ts.map +1 -0
- package/lib/types/scorecard.js +186 -0
- package/lib/types/scorecard.js.map +1 -0
- package/lib/types/service.d.ts +32 -2
- package/lib/types/service.d.ts.map +1 -1
- package/lib/types/service.js.map +1 -1
- package/lib/types/store.d.ts +21 -1
- package/lib/types/store.d.ts.map +1 -1
- package/lib/types/store.js +13 -1
- package/lib/types/store.js.map +1 -1
- package/lib/types/tools/clean.d.ts.map +1 -1
- package/lib/types/tools/clean.js +76 -1
- package/lib/types/tools/clean.js.map +1 -1
- package/lib/types/tools/profile-report-schema.d.ts +315 -0
- package/lib/types/tools/profile-report-schema.d.ts.map +1 -0
- package/lib/types/tools/profile-report-schema.js +95 -0
- package/lib/types/tools/profile-report-schema.js.map +1 -0
- package/lib/types/tools/profile.d.ts.map +1 -1
- package/lib/types/tools/profile.js +6 -53
- package/lib/types/tools/profile.js.map +1 -1
- package/lib/types/tools/report.d.ts +14 -0
- package/lib/types/tools/report.d.ts.map +1 -0
- package/lib/types/tools/report.js +86 -0
- package/lib/types/tools/report.js.map +1 -0
- package/lib/types/tools/shared.d.ts.map +1 -1
- package/lib/types/tools/shared.js +22 -2
- package/lib/types/tools/shared.js.map +1 -1
- package/lib/types/tools/verify.d.ts.map +1 -1
- package/lib/types/tools/verify.js +36 -1
- package/lib/types/tools/verify.js.map +1 -1
- package/lib/types/verify.d.ts +49 -3
- package/lib/types/verify.d.ts.map +1 -1
- package/lib/types/verify.js +111 -4
- package/lib/types/verify.js.map +1 -1
- package/lib/types/version.d.ts +1 -1
- package/lib/types/version.js +1 -1
- package/package.json +1 -1
- package/src/config.ts +52 -0
- package/src/contract.ts +190 -0
- package/src/dataset.ts +101 -21
- package/src/events.ts +2 -0
- package/src/index.ts +25 -8
- package/src/presets.ts +146 -0
- package/src/profile.ts +101 -11
- package/src/provider-local.ts +83 -5
- package/src/scorecard.ts +244 -0
- package/src/service.ts +34 -2
- package/src/store.ts +28 -2
- package/src/tools/clean.ts +76 -1
- package/src/tools/profile-report-schema.ts +97 -0
- package/src/tools/profile.ts +8 -56
- package/src/tools/report.ts +108 -0
- package/src/tools/shared.ts +21 -2
- package/src/tools/verify.ts +40 -2
- package/src/verify.ts +151 -5
- package/src/version.ts +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,38 @@ All notable changes to this project are documented in this file.
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
|
6
6
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
7
|
|
|
8
|
+
## [0.2.0] - 2026-08-23
|
|
9
|
+
|
|
10
|
+
### Added
|
|
11
|
+
|
|
12
|
+
- DAMA six-dimension quality scorecard in `data_profile`: completeness, uniqueness, validity, consistency, timeliness, and accuracy (limited declared-schema definition; accuracy is reported undetermined without a declared schema rather than fabricated). The scorecard flows through the profile report, the persisted storage-domain record, and the tool result.
|
|
13
|
+
- Full-row sha256 content-hash duplicate detection in `data_profile`, reporting the duplicate rate plus a bounded sample of 0-based duplicate row indexes (capped by `evidenceRowLimit`).
|
|
14
|
+
- Pre-delivery contract validation summary on `data_clean` (dedupe before/after row comparison, uniqueness regression, non-null and type regressions) plus a persisted clean before/after profile diff report under the new `clean-diff` storage record kind.
|
|
15
|
+
- `data_report` tool plus `ctx.dataQuality.getReport`/`listReports` query methods to read persisted reports back by `reportKey` (path-safe validation, missing records fail loud) or by `kind`, ordered chronologically.
|
|
16
|
+
- `data_clean` `dryRun` parameter: no file written and no report persisted, returning the per-column cleaning plan (in `contract.columnDecisions`) plus the expected `contract`/`diffPreview`.
|
|
17
|
+
- Numeric distribution `count`/`distinct` fields on numeric column profiles.
|
|
18
|
+
- Config `scorecardWeights` (six per-dimension weights, non-negative, equal by default) driving a `weightedOverall` scorecard total.
|
|
19
|
+
- Per-column decision trace in the clean contract (`columnDecisions`: strategy + affected rows), persisted with the clean report.
|
|
20
|
+
- File-encoding reporting in `data_profile` (`encoding`: UTF-8 BOM presence and validity; invalid UTF-8 is reported, not fatal).
|
|
21
|
+
- Built-in industry profile presets (`retail`, `saas`, `fund`, `real-estate`, `e-commerce`, `healthcare`, `logistics`, `manufacturing`, `energy`) via the `data_profile` `industryPreset` parameter, feeding the scorecard `accuracy` dimension; unknown ids fail loud.
|
|
22
|
+
- Metric reconciliation on `data_verify`: optional `expectations` (rowCount/columnSum/columnMean/uniqueCount/nullCount, each with `column`, `expected`, optional relative `tolerance`) reconcile deterministic computed values; a mismatch is a normal `passed: false` verdict with actual/expected/tolerance detail, while invalid metrics, columns, and out-of-range tolerances fail loud. Results flow into the verify report and its persisted record.
|
|
23
|
+
|
|
24
|
+
### Deviations
|
|
25
|
+
|
|
26
|
+
Documented, deliberate non-goals for this repository (recorded rather than silently omitted):
|
|
27
|
+
|
|
28
|
+
- **Native xlsx** — a real Excel parser needs a new dependency (or a sidecar); this repo keeps a zero-dependency parsing contract, so CSV/TSV/JSON/JSONL are the supported dataset formats, now with UTF-8/BOM encoding detection.
|
|
29
|
+
- **SQL / DuckDB access** — database drivers are out of scope for the zero-dependency, in-process TypeScript seam; tabular datasets are files.
|
|
30
|
+
- **`qsv` as an alternative parser** — an external binary violates the no-external-processes contract.
|
|
31
|
+
- **Slot client report panel** — a browser report panel needs a client half and bundle changes; the consumption path is covered by the `data_report` tool plus storage-domain persistence instead.
|
|
32
|
+
- **Computer-use fallback collection** — macOS-only and brittle; the regular filesystem capability already covers the supported collection path.
|
|
33
|
+
|
|
34
|
+
## [0.1.3] - 2026-08-23
|
|
35
|
+
|
|
36
|
+
### Changed
|
|
37
|
+
|
|
38
|
+
- Persist the bounded clean `preview` in the durable `clean` report so the model-visible canonical result is fully reconstructable from its `reportKey` (model-visible ⟺ logged).
|
|
39
|
+
|
|
8
40
|
## [0.1.2] - 2026-08-22
|
|
9
41
|
|
|
10
42
|
### Changed
|
package/README.es.md
CHANGED
|
@@ -21,7 +21,7 @@ Todo el cálculo es TypeScript puro dentro del proceso del harness — el modelo
|
|
|
21
21
|
- **Herramienta `data_profile`** — perfilado de datasets: conteos de filas/columnas, tipos de columna inferidos (number/date/boolean/string/empty/mixed), tasas de valores faltantes, conteos de valores únicos, distribuciones numéricas (min/max/mean/median/p25/p75), conteo de valores atípicos IQR, notas de sospecha por tipos mixtos y conteo de filas duplicadas de la tabla completa. Muestreo sistemático determinista opcional para archivos grandes.
|
|
22
22
|
- **Herramienta `data_clean`** — reglas declarativas de limpieza en orden: `dedupe` (por grupo de columnas), `fill-missing` (constant/mean/median/forward), `coerce-type` (number/date/boolean; los fallos se cuentan y quedan como faltantes), `normalize-unit` (p. ej. sufijos 万/亿 a unidades base), `trim`, `map-values` (mapeo de enumeraciones). Devuelve un registro de auditoría por regla más una vista previa acotada; solo escribe el dataset limpio cuando se indica `outputPath` y nunca sobrescribe el origen.
|
|
23
23
|
- **Herramienta `data_verify`** — reglas declarativas de verificación: `not-null`, `unique`, `range`, `regex`, `enum`, `cross-column` (p. ej. `startDate < endDate`), `freshness` (columna de fecha dentro de N días de una fecha de referencia). pass/fail por regla con evidencia acotada de filas fallidas; un fallo global es un resultado normal `passed: false`, no un error de herramienta.
|
|
24
|
-
- **Informes duraderos** — cada ejecución de perfilado/limpieza/verificación/citas persiste en el dominio de almacenamiento `data_quality` (backend JSON), con clave de marca de tiempo más huella de la ruta del dataset; la clave se devuelve como `reportKey` en los resultados.
|
|
24
|
+
- **Informes duraderos** — cada ejecución de perfilado/limpieza/verificación/citas persiste en el dominio de almacenamiento `data_quality` (backend JSON), con clave de marca de tiempo más huella de la ruta del dataset; la clave se devuelve como `reportKey` en los resultados. Los informes de limpieza también persisten la vista previa acotada, de modo que todo resultado visible para el modelo es reconstruible a partir de su `reportKey`.
|
|
25
25
|
- **Eventos de sesión** — en hosts que los soportan con seguridad, las ejecuciones añaden eventos `data-quality/profile` / `data-quality/clean` / `data-quality/verify` (con la marca `ignorable` donde se admite). En 0.1.1-rc.2 el append se omite por diseño — el informe del dominio de almacenamiento es siempre la copia duradera (véase «Known limitations»).
|
|
26
26
|
|
|
27
27
|
## Quick start
|
|
@@ -77,6 +77,7 @@ Todas las claves son opcionales (se muestran los valores por defecto); los valor
|
|
|
77
77
|
| `allowedExtensions` | `['.csv', '.tsv', '.json', '.jsonl']` | Extensiones aceptadas como datasets. |
|
|
78
78
|
| `workspaceRoot` | `""` | Raíz absoluta para llamadas a nivel de SERVICIO (p. ej. `verifyCitations`) sin workspace de sesión; vacío = directorio de arranque del proceso del harness. Las herramientas siempre usan el cwd del workspace de la sesión. |
|
|
79
79
|
| `storeReports` | `true` | Persistir los informes en el dominio de almacenamiento `data_quality` y devolver `reportKey`. |
|
|
80
|
+
| `scorecardWeights` | todo 1 (igual) | Ponderaciones por dimensión (completeness/uniqueness/validity/consistency/timeliness/accuracy) para el total ponderado del scorecard; cada peso debe ser un número no negativo. |
|
|
80
81
|
|
|
81
82
|
## Tools & surfaces
|
|
82
83
|
|
|
@@ -170,7 +171,12 @@ pnpm run verify:self-contained && pnpm run verify:artifacts && pnpm run verify:r
|
|
|
170
171
|
|
|
171
172
|
## Contributors
|
|
172
173
|
|
|
173
|
-
|
|
174
|
+
Gracias a todas las personas que han dado forma a este plugin.
|
|
175
|
+
|
|
176
|
+
- **PerryLink** — mantenimiento y releases (`0.1.2`/`0.1.3`), mejoras de peer-dependencies, las insignias de npm version/downloads/CI, y correcciones recientes.
|
|
177
|
+
- **dsh-data-quality contributors** — el scaffold inicial, el seam `ctx.dataQuality` y el contrato congelado `verifyCitations`, la capa de datasets determinista y los motores puros, los informes del dominio de almacenamiento `data_quality`, la suite vitest con servicios reales, los flujos CI/compat/release, y los READMEs en cinco idiomas.
|
|
178
|
+
|
|
179
|
+
Este repositorio aún no tiene historial público de issues o pull requests; aquí se acreditarán los números de PR/issue cuando lleguen.
|
|
174
180
|
|
|
175
181
|
## PerryLink DSH Plugin Family
|
|
176
182
|
|
package/README.hi.md
CHANGED
|
@@ -21,7 +21,7 @@
|
|
|
21
21
|
- **`data_profile` टूल** — डेटासेट प्रोफ़ाइलिंग: पंक्ति/स्तंभ गिनती, अनुमानित स्तंभ प्रकार (number/date/boolean/string/empty/mixed), अनुपस्थिति दर, अद्वितीय मान संख्या, संख्यात्मक वितरण (min/max/mean/median/p25/p75), IQR आउटलायर गिनती, मिश्रित-प्रकार संदेह टिप्पणियाँ, और पूरी तालिका की डुप्लिकेट पंक्ति गिनती। बड़ी फ़ाइलों के लिए वैकल्पिक निर्धारणात्मक व्यवस्थित प्रतिचयन।
|
|
22
22
|
- **`data_clean` टूल** — क्रमबद्ध घोषणात्मक क्लीनिंग नियम: `dedupe` (स्तंभ समूह से), `fill-missing` (constant/mean/median/forward), `coerce-type` (number/date/boolean; विफलताएँ गिनी जाती हैं और अनुपस्थित बन जाती हैं), `normalize-unit` (जैसे 万/亿 प्रत्यय को आधार इकाई में), `trim`, `map-values` (एनुम मैपिंग)। प्रति-नियम ऑडिट लॉग और सीमित पूर्वावलोकन लौटाता है; केवल `outputPath` दिए जाने पर साफ़ डेटासेट लिखता है और मूल फ़ाइल को कभी अधिलेखित नहीं करता।
|
|
23
23
|
- **`data_verify` टूल** — घोषणात्मक सत्यापन नियम: `not-null`, `unique`, `range`, `regex`, `enum`, `cross-column` (जैसे `startDate < endDate`), `freshness` (संदर्भ तिथि से N दिनों के भीतर तिथि स्तंभ)। प्रति-नियम pass/fail और सीमित असफल-पंक्ति साक्ष्य; समग्र असफलता सामान्य `passed: false` परिणाम है, टूल त्रुटि नहीं।
|
|
24
|
-
- **टिकाऊ रिपोर्ट** — हर प्रोफ़ाइल/क्लीन/सत्यापन/उद्धरण रन `data_quality` स्टोरेज डोमेन (JSON बैकएंड) में सहेजा जाता है, कुंजी = रन टाइमस्टैम्प + डेटासेट-पथ फ़िंगरप्रिंट; कुंजी टूल परिणामों में `reportKey` के रूप में लौटती है।
|
|
24
|
+
- **टिकाऊ रिपोर्ट** — हर प्रोफ़ाइल/क्लीन/सत्यापन/उद्धरण रन `data_quality` स्टोरेज डोमेन (JSON बैकएंड) में सहेजा जाता है, कुंजी = रन टाइमस्टैम्प + डेटासेट-पथ फ़िंगरप्रिंट; कुंजी टूल परिणामों में `reportKey` के रूप में लौटती है। क्लीन रिपोर्ट सीमित पूर्वावलोकन भी सहेजती हैं, इसलिए हर मॉडल-दृश्य परिणाम केवल `reportKey` से पुनर्निर्मित किया जा सकता है।
|
|
25
25
|
- **सत्र ईवेंट** — जिन होस्ट पर सुरक्षित रूप से संभव है, रन `data-quality/profile` / `data-quality/clean` / `data-quality/verify` ईवेंट जोड़ते हैं (जहाँ समर्थित हो वहाँ `ignorable` चिह्न सहित)। 0.1.1-rc.2 पर append जानबूझकर छोड़ा जाता है — स्टोरेज-डोमेन रिपोर्ट हमेशा टिकाऊ प्रति होती है (देखें «Known limitations»)।
|
|
26
26
|
|
|
27
27
|
## Quick start
|
|
@@ -77,6 +77,7 @@ dsh plugin --profile web remove dsh-data-quality # अनइंस्टॉल
|
|
|
77
77
|
| `allowedExtensions` | `['.csv', '.tsv', '.json', '.jsonl']` | डेटासेट के रूप में स्वीकृत एक्सटेंशन। |
|
|
78
78
|
| `workspaceRoot` | `""` | SERVICE-स्तरीय कॉल (जैसे `verifyCitations`) के लिए निरपेक्ष रूट जिनके पास सत्र वर्कस्पेस नहीं है; रिक्त = harness प्रक्रिया का आरंभ निर्देशिका। टूल कॉल हमेशा सत्र वर्कस्पेस cwd उपयोग करती हैं। |
|
|
79
79
|
| `storeReports` | `true` | रन रिपोर्ट `data_quality` स्टोरेज डोमेन में सहेजें और `reportKey` लौटाएँ। |
|
|
80
|
+
| `scorecardWeights` | सभी 1 (बराबर) | स्कोरकार्ड के भारित कुल के लिए प्रति-आयाम भार (completeness/uniqueness/validity/consistency/timeliness/accuracy); हर भार एक गैर-ऋणात्मक संख्या होना चाहिए। |
|
|
80
81
|
|
|
81
82
|
## Tools & surfaces
|
|
82
83
|
|
|
@@ -170,7 +171,12 @@ pnpm run verify:self-contained && pnpm run verify:artifacts && pnpm run verify:r
|
|
|
170
171
|
|
|
171
172
|
## Contributors
|
|
172
173
|
|
|
173
|
-
|
|
174
|
+
इस प्लगिन को आकार देने वाले सभी का धन्यवाद।
|
|
175
|
+
|
|
176
|
+
- **PerryLink** — रखरखाव और रिलीज़ (`0.1.2`/`0.1.3`), peer-dependencies उन्नयन, npm version/downloads/CI बैज, और हालिया सुधार।
|
|
177
|
+
- **dsh-data-quality contributors** — प्रारंभिक scaffold, `ctx.dataQuality` सीम और फ्रोज़न `verifyCitations` अनुबंध, नियतात्मक डेटासेट परत और शुद्ध इंजन, `data_quality` स्टोरेज-डोमेन रिपोर्ट, वास्तविक-सेवा vitest सुइट, CI/compat/release वर्कफ़्लो, और पाँच-भाषा README।
|
|
178
|
+
|
|
179
|
+
इस रिपॉजिटरी का अभी कोई सार्वजनिक issue या pull request इतिहास नहीं है; आने पर PR/issue संख्याएँ यहाँ अंकित की जाएँगी।
|
|
174
180
|
|
|
175
181
|
## PerryLink DSH Plugin Family
|
|
176
182
|
|
package/README.md
CHANGED
|
@@ -24,10 +24,11 @@ All computation is plain TypeScript in the harness process — the model never d
|
|
|
24
24
|
## What you get
|
|
25
25
|
|
|
26
26
|
- **`ctx.dataQuality` service** — a Cordis service other plugins may optionally consume (`inject = ['dataQuality']`). Besides the three dataset operations behind the tools, it implements the frozen `verifyCitations(request)` contract: verify that numbers/strings cited in a document match a dataset snapshot, with relative-tolerance numeric comparison and `verified` / `mismatch` / `not-found` / `unverifiable` statuses.
|
|
27
|
-
- **`data_profile` tool** — dataset profiling: row/column counts, inferred column types (number/date/boolean/string/empty/mixed), missing rates, unique counts, numeric distributions (min/max/mean/median/p25/p75), IQR outlier counts, mixed-type suspicion notes, and full-table duplicate
|
|
28
|
-
- **`data_clean` tool** — ordered declarative cleaning rules: `dedupe` (by column group), `fill-missing` (constant/mean/median/forward), `coerce-type` (number/date/boolean; failures counted and set to missing), `normalize-unit` (e.g. 万/亿 suffixes to base units), `trim`, `map-values` (enum mapping). Returns a per-rule audit log
|
|
27
|
+
- **`data_profile` tool** — dataset profiling: row/column counts, inferred column types (number/date/boolean/string/empty/mixed), missing rates, unique counts, numeric distributions (min/max/mean/median/p25/p75), IQR outlier counts, mixed-type suspicion notes, and full-table sha256 content-hash duplicate detection with the duplicate rate and a bounded sample of duplicate row indexes. Adds a deterministic DAMA six-dimension scorecard (completeness, uniqueness, validity, consistency, timeliness, accuracy — accuracy is reported undetermined without a declared schema, never fabricated). Optional deterministic systematic sampling for large files.
|
|
28
|
+
- **`data_clean` tool** — ordered declarative cleaning rules: `dedupe` (by column group), `fill-missing` (constant/mean/median/forward), `coerce-type` (number/date/boolean; failures counted and set to missing), `normalize-unit` (e.g. 万/亿 suffixes to base units), `trim`, `map-values` (enum mapping). Returns a per-rule audit log, a pre-delivery contract validation summary (dedupe before/after, uniqueness, non-null and type regressions), and a bounded preview; writes the cleaned dataset only when `outputPath` is given, and never overwrites the source.
|
|
29
29
|
- **`data_verify` tool** — declarative verification rules: `not-null`, `unique`, `range`, `regex`, `enum`, `cross-column` (e.g. `startDate < endDate`), `freshness` (date column within N days of a reference date). Per-rule pass/fail with capped failing-row evidence; an overall failure is a normal `passed: false` result, not a tool error.
|
|
30
|
-
-
|
|
30
|
+
- **`data_report` tool** — read persisted reports back from the `data_quality` storage domain: by exact `reportKey` (path-safe validation, missing keys fail loud) or by `kind` (chronological listing). Returns the report envelope(s) — kind, dataset, timestamp, and the full stored report.
|
|
31
|
+
- **Durable reports** — every profile/clean/verify/citation run persists to the `data_quality` storage domain (JSON backend), keyed by run timestamp plus a dataset-path fingerprint; the key is returned as `reportKey` in tool results. Clean reports also persist the bounded preview and the contract summary, so every model-visible result is reconstructable from its `reportKey`; each clean run additionally persists a `clean-diff` before/after profile report.
|
|
31
32
|
- **Session events** — on hosts that can carry them safely, runs append `data-quality/profile` / `data-quality/clean` / `data-quality/verify` events (with the `ignorable` marker where supported). On 0.1.1-rc.2 the append is skipped by design — the storage-domain report is always the durable copy (see "Known limitations").
|
|
32
33
|
|
|
33
34
|
## Quick start
|
|
@@ -83,14 +84,15 @@ All keys are optional (defaults shown); invalid values fail loudly at load. Ever
|
|
|
83
84
|
| `allowedExtensions` | `['.csv', '.tsv', '.json', '.jsonl']` | Extensions accepted as datasets. |
|
|
84
85
|
| `workspaceRoot` | `""` | Absolute root for SERVICE-level calls (e.g. `verifyCitations`) that carry no session workspace; empty = the harness process launch directory. Tool calls always use the session's workspace cwd. |
|
|
85
86
|
| `storeReports` | `true` | Persist run reports to the `data_quality` storage domain and return `reportKey`. |
|
|
87
|
+
| `scorecardWeights` | all 1 (equal) | Per-dimension weights (completeness/uniqueness/validity/consistency/timeliness/accuracy) for the scorecard's weighted overall total; each weight must be a non-negative number. |
|
|
86
88
|
|
|
87
89
|
## Tools & surfaces
|
|
88
90
|
|
|
89
|
-
### `data_profile({ path, sample? })`
|
|
91
|
+
### `data_profile({ path, sample?, industryPreset? })`
|
|
90
92
|
|
|
91
|
-
Profiles a workspace dataset. `path` is workspace-relative (`.csv`/`.tsv`/`.json`/`.jsonl`; JSON must be an array of flat objects). `sample` takes every `ceil(N/sample)`-th row for the column cards (deterministic; row counts stay exact). Returns the structured report
|
|
93
|
+
Profiles a workspace dataset. `path` is workspace-relative (`.csv`/`.tsv`/`.json`/`.jsonl`; JSON must be an array of flat objects). `sample` takes every `ceil(N/sample)`-th row for the column cards (deterministic; row counts stay exact). `industryPreset` (`retail`/`saas`/`fund`/`real-estate`/`e-commerce`/`healthcare`/`logistics`/`manufacturing`/`energy`) injects that industry's expected columns so the scorecard `accuracy` dimension becomes determinable; unknown ids fail loud. Returns the structured report — the duplicate rate, bounded duplicate-row indexes, numeric `count`/`distinct` distributions, file `encoding` (UTF-8 BOM + validity), and the weighted six-dimension DAMA scorecard — and renders a human-readable per-column summary plus scorecard.
|
|
92
94
|
|
|
93
|
-
### `data_clean({ path, rules, outputPath? })`
|
|
95
|
+
### `data_clean({ path, rules, outputPath?, dryRun? })`
|
|
94
96
|
|
|
95
97
|
Applies `rules` in array order, each seeing the previous rule's output. Rule reference:
|
|
96
98
|
|
|
@@ -103,9 +105,13 @@ Applies `rules` in array order, each seeing the previous rule's output. Rule ref
|
|
|
103
105
|
| `trim` | `columns?` | Trim whitespace of string cells (all columns when omitted). |
|
|
104
106
|
| `map-values` | `column`, `map`, `else?` | Exact-match mapping; unmapped values stay (`keep`, default) or become `missing`. |
|
|
105
107
|
|
|
106
|
-
The source file is **never** overwritten. With `outputPath` the cleaned dataset is written there (workspace-confined, format by extension); without it the run is preview-only.
|
|
108
|
+
The source file is **never** overwritten. With `outputPath` the cleaned dataset is written there (workspace-confined, format by extension); without it the run is preview-only. With `dryRun: true` no file is written and nothing is persisted — the result returns the per-column cleaning plan and the expected `contract`/`diffPreview`. The result also carries a pre-delivery `contract` summary (dedupe before/after rows, uniqueness over the dedupe key or full rows, non-null regression for `fill-missing` columns, type regression for `coerce-type` columns, and the per-column decision trace), and a `clean-diff` before/after profile report is persisted to the storage domain.
|
|
107
109
|
|
|
108
|
-
### `
|
|
110
|
+
### `data_report({ key?, kind? })`
|
|
111
|
+
|
|
112
|
+
Reads persisted reports back from the storage domain. Pass `key` (the exact `reportKey` a prior run returned) to fetch one report, or `kind` (`profile`/`clean`/`clean-diff`/`verify`/`citations`) to list every report of that kind chronologically; exactly one of `key`/`kind` is required. Malformed or missing keys fail loud.
|
|
113
|
+
|
|
114
|
+
### `data_verify({ path, rules, expectations? })`
|
|
109
115
|
|
|
110
116
|
Evaluates verification rules. Rule reference:
|
|
111
117
|
|
|
@@ -119,6 +125,8 @@ Evaluates verification rules. Rule reference:
|
|
|
119
125
|
| `cross-column` | `left`, `op`, `rightColumn?`, `value?` | Compare per row: numeric when both sides parse, dates compare as epochs, strings only for `==`/`!=` (exactly one of `rightColumn`/`value`). |
|
|
120
126
|
| `freshness` | `column`, `maxAgeDays`, `asOf?` | Fail dates older than `maxAgeDays` before `asOf` (default: now); unparseable/missing fails. |
|
|
121
127
|
|
|
128
|
+
`expectations` reconciles deterministic metrics against expected values: `rowCount`, `columnSum`, `columnMean`, `uniqueCount`, `nullCount` (each with `column` except `rowCount`, plus `expected` and an optional relative `tolerance` in [0, 1]). Each expectation yields `passed` plus `actual`/`expected`/`tolerance`; a mismatch is a normal `passed: false` verdict, never a tool error. Invalid metrics, missing columns, and out-of-range tolerances fail loud.
|
|
129
|
+
|
|
122
130
|
A missing cell fails every rule that reads it. Evidence is capped at `evidenceRowLimit` failing rows per rule.
|
|
123
131
|
|
|
124
132
|
### `ctx.dataQuality` (for other plugins)
|
|
@@ -176,7 +184,12 @@ pnpm run verify:self-contained && pnpm run verify:artifacts && pnpm run verify:r
|
|
|
176
184
|
|
|
177
185
|
## Contributors
|
|
178
186
|
|
|
179
|
-
|
|
187
|
+
Thanks to everyone who has shaped this plugin.
|
|
188
|
+
|
|
189
|
+
- **PerryLink** — maintenance and releases (`0.1.2`/`0.1.3`), peer-dependency upgrades, the npm version/downloads/CI badges, and recent fixes.
|
|
190
|
+
- **dsh-data-quality contributors** — the initial scaffold, the `ctx.dataQuality` seam and frozen `verifyCitations` contract, the deterministic dataset layer and pure engines, the `data_quality` storage-domain reports, the real-service vitest suite, the CI/compat/release workflows, and the five-language READMEs.
|
|
191
|
+
|
|
192
|
+
This repository has no public issue or pull request history yet; individual PR/issue numbers will be credited here as they arrive.
|
|
180
193
|
|
|
181
194
|
## PerryLink DSH Plugin Family
|
|
182
195
|
|
package/README.pt.md
CHANGED
|
@@ -21,7 +21,7 @@ Todo o cálculo é TypeScript puro no processo do harness — o modelo nunca faz
|
|
|
21
21
|
- **Ferramenta `data_profile`** — perfilamento de datasets: contagens de linhas/colunas, tipos de coluna inferidos (number/date/boolean/string/empty/mixed), taxas de ausência, contagens de valores únicos, distribuições numéricas (min/max/mean/median/p25/p75), contagem de outliers IQR, notas de suspeita de tipos mistos e contagem de linhas duplicadas da tabela inteira. Amostragem sistemática determinística opcional para arquivos grandes.
|
|
22
22
|
- **Ferramenta `data_clean`** — regras declarativas de limpeza em ordem: `dedupe` (por grupo de colunas), `fill-missing` (constant/mean/median/forward), `coerce-type` (number/date/boolean; falhas contadas e viram ausentes), `normalize-unit` (p. ex. sufixos 万/亿 para unidades base), `trim`, `map-values` (mapeamento de enumerações). Retorna um log de auditoria por regra mais uma prévia limitada; só grava o dataset limpo quando `outputPath` é dado e nunca sobrescreve a origem.
|
|
23
23
|
- **Ferramenta `data_verify`** — regras declarativas de verificação: `not-null`, `unique`, `range`, `regex`, `enum`, `cross-column` (p. ex. `startDate < endDate`), `freshness` (coluna de data dentro de N dias de uma data de referência). pass/fail por regra com evidência limitada de linhas falhas; uma falha geral é um resultado normal `passed: false`, não um erro de ferramenta.
|
|
24
|
-
- **Relatórios duráveis** — cada execução de perfilamento/limpeza/verificação/citações persiste no domínio de armazenamento `data_quality` (backend JSON), com chave de timestamp mais impressão digital do caminho do dataset; a chave é retornada como `reportKey` nos resultados.
|
|
24
|
+
- **Relatórios duráveis** — cada execução de perfilamento/limpeza/verificação/citações persiste no domínio de armazenamento `data_quality` (backend JSON), com chave de timestamp mais impressão digital do caminho do dataset; a chave é retornada como `reportKey` nos resultados. Os relatórios de limpeza também persistem a pré-visualização limitada, de modo que todo resultado visível ao modelo é reconstruível a partir do seu `reportKey`.
|
|
25
25
|
- **Eventos de sessão** — em hosts que os suportam com segurança, as execuções anexam eventos `data-quality/profile` / `data-quality/clean` / `data-quality/verify` (com a marca `ignorable` onde suportado). Em 0.1.1-rc.2 o append é omitido por design — o relatório do domínio de armazenamento é sempre a cópia durável (ver «Known limitations»).
|
|
26
26
|
|
|
27
27
|
## Quick start
|
|
@@ -77,6 +77,7 @@ Todas as chaves são opcionais (valores padrão mostrados); valores inválidos f
|
|
|
77
77
|
| `allowedExtensions` | `['.csv', '.tsv', '.json', '.jsonl']` | Extensões aceitas como datasets. |
|
|
78
78
|
| `workspaceRoot` | `""` | Raiz absoluta para chamadas de nível de SERVIÇO (p. ex. `verifyCitations`) sem workspace de sessão; vazio = diretório de arranque do processo do harness. Ferramentas sempre usam o cwd do workspace da sessão. |
|
|
79
79
|
| `storeReports` | `true` | Persistir relatórios no domínio de armazenamento `data_quality` e retornar `reportKey`. |
|
|
80
|
+
| `scorecardWeights` | todo 1 (igual) | Pesos por dimensão (completeness/uniqueness/validity/consistency/timeliness/accuracy) para o total ponderado do scorecard; cada peso deve ser um número não negativo. |
|
|
80
81
|
|
|
81
82
|
## Tools & surfaces
|
|
82
83
|
|
|
@@ -170,7 +171,12 @@ pnpm run verify:self-contained && pnpm run verify:artifacts && pnpm run verify:r
|
|
|
170
171
|
|
|
171
172
|
## Contributors
|
|
172
173
|
|
|
173
|
-
|
|
174
|
+
Obrigado a todos os que deram forma a este plugin.
|
|
175
|
+
|
|
176
|
+
- **PerryLink** — manutenção e releases (`0.1.2`/`0.1.3`), atualizações de peer-dependencies, os selos npm version/downloads/CI, e correções recentes.
|
|
177
|
+
- **dsh-data-quality contributors** — o scaffold inicial, o seam `ctx.dataQuality` e o contrato congelado `verifyCitations`, a camada de datasets determinística e os motores puros, os relatórios do domínio de armazenamento `data_quality`, a suíte vitest com serviços reais, os fluxos CI/compat/release, e os READMEs em cinco línguas.
|
|
178
|
+
|
|
179
|
+
Este repositório ainda não tem histórico público de issues ou pull requests; os números de PR/issue serão creditados aqui quando surgirem.
|
|
174
180
|
|
|
175
181
|
## PerryLink DSH Plugin Family
|
|
176
182
|
|
package/README.zh.md
CHANGED
|
@@ -21,7 +21,7 @@
|
|
|
21
21
|
- **`data_profile` 工具** —— 数据集梳理:行/列数、推断列类型(number/date/boolean/string/empty/mixed)、缺失率、唯一值数、数值分布(min/max/mean/median/p25/p75)、IQR 离群计数、混合类型疑点提示、全表重复行计数。大文件可选确定性系统采样。
|
|
22
22
|
- **`data_clean` 工具** —— 有序声明式清洗规则:`dedupe`(按列组)、`fill-missing`(常量/均值/中位数/前向填充)、`coerce-type`(number/date/boolean,失败计数并置缺失)、`normalize-unit`(万/亿 等单位后缀归一)、`trim`、`map-values`(枚举映射)。返回逐规则审计日志与有界预览;仅在给出 `outputPath` 时落盘,且绝不覆盖源文件。
|
|
23
23
|
- **`data_verify` 工具** —— 声明式核查规则:`not-null`、`unique`、`range`、`regex`、`enum`、`cross-column`(如 `startDate < endDate`)、`freshness`(日期列距参考日期不超过 N 天)。逐规则 pass/fail 并附有界失败行证据;整体失败是正常结果 `passed: false`,不是工具错误。
|
|
24
|
-
- **持久化报告** —— 每次梳理/清洗/核查/引用检查都写入 `data_quality` storage domain(JSON 后端),键为运行时间戳加数据集路径指纹;工具结果以 `reportKey`
|
|
24
|
+
- **持久化报告** —— 每次梳理/清洗/核查/引用检查都写入 `data_quality` storage domain(JSON 后端),键为运行时间戳加数据集路径指纹;工具结果以 `reportKey` 返回该键。清洗报告同时持久化有界预览,因此任何模型可见结果都能仅凭 `reportKey` 重建。
|
|
25
25
|
- **会话事件** —— 宿主支持时,运行会追加 `data-quality/profile` / `data-quality/clean` / `data-quality/verify` 事件(支持处带 `ignorable` 标记)。在 0.1.1-rc.2 上按设计跳过 append —— storage domain 报告始终是持久副本(见「Known limitations」)。
|
|
26
26
|
|
|
27
27
|
## Quick start
|
|
@@ -77,6 +77,7 @@ dsh plugin --profile web remove dsh-data-quality # 卸载
|
|
|
77
77
|
| `allowedExtensions` | `['.csv', '.tsv', '.json', '.jsonl']` | 可作为数据集的扩展名。 |
|
|
78
78
|
| `workspaceRoot` | `""` | 服务级调用(如 `verifyCitations`)无会话工作区时使用的绝对根;空 = harness 进程启动目录。工具调用始终使用会话工作区 cwd。 |
|
|
79
79
|
| `storeReports` | `true` | 把运行报告写入 `data_quality` storage domain 并返回 `reportKey`。 |
|
|
80
|
+
| `scorecardWeights` | 全为 1(等权) | 六维加权总分的各维度权重(completeness/uniqueness/validity/consistency/timeliness/accuracy);每个权重必须是非负数。 |
|
|
80
81
|
|
|
81
82
|
## Tools & surfaces
|
|
82
83
|
|
|
@@ -170,7 +171,12 @@ pnpm run verify:self-contained && pnpm run verify:artifacts && pnpm run verify:r
|
|
|
170
171
|
|
|
171
172
|
## Contributors
|
|
172
173
|
|
|
173
|
-
|
|
174
|
+
感谢所有塑造本插件的贡献者。
|
|
175
|
+
|
|
176
|
+
- **PerryLink** — 维护与发布(`0.1.2`/`0.1.3`)、peer 依赖升级、npm 版本/下载量/CI 徽章,以及近期修复。
|
|
177
|
+
- **dsh-data-quality contributors** — 初始脚手架、`ctx.dataQuality` 能力缝与冻结的 `verifyCitations` 契约、确定性数据集层与纯引擎、`data_quality` storage-domain 报告、真实服务 vitest 套件、CI/compat/release 工作流,以及五语 README。
|
|
178
|
+
|
|
179
|
+
本仓库暂无公开 issue 或 pull request 历史;后续将在此逐个记名 PR/issue 编号。
|
|
174
180
|
|
|
175
181
|
## PerryLink DSH Plugin Family
|
|
176
182
|
|
package/cordis.patch.yml
CHANGED
|
@@ -44,3 +44,14 @@
|
|
|
44
44
|
# Persist profile/clean/verify reports to the `data_quality` storage
|
|
45
45
|
# domain (durable, queryable). Keys are returned as `reportKey`.
|
|
46
46
|
storeReports: true
|
|
47
|
+
# Per-dimension weights for the scorecard's weighted overall total
|
|
48
|
+
# (completeness/uniqueness/validity/consistency/timeliness/accuracy).
|
|
49
|
+
# Each weight must be a non-negative number; omit a dimension for the
|
|
50
|
+
# equal-weight default of 1.
|
|
51
|
+
scorecardWeights:
|
|
52
|
+
completeness: 1
|
|
53
|
+
uniqueness: 1
|
|
54
|
+
validity: 1
|
|
55
|
+
consistency: 1
|
|
56
|
+
timeliness: 1
|
|
57
|
+
accuracy: 1
|