dsh-data-quality 0.1.3 → 0.2.0

package/CHANGELOG.md

@@ -5,6 +5,32 @@ All notable changes to this project are documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.2.0] - 2026-08-23
+
+### Added
+
+- DAMA six-dimension quality scorecard in `data_profile`: completeness, uniqueness, validity, consistency, timeliness, and accuracy (limited declared-schema definition; accuracy is reported undetermined without a declared schema rather than fabricated). The scorecard flows through the profile report, the persisted storage-domain record, and the tool result.
+- Full-row sha256 content-hash duplicate detection in `data_profile`, reporting the duplicate rate plus a bounded sample of 0-based duplicate row indexes (capped by `evidenceRowLimit`).
+- Pre-delivery contract validation summary on `data_clean` (dedupe before/after row comparison, uniqueness regression, non-null and type regressions) plus a persisted clean before/after profile diff report under the new `clean-diff` storage record kind.
+- `data_report` tool plus `ctx.dataQuality.getReport`/`listReports` query methods to read persisted reports back by `reportKey` (path-safe validation, missing records fail loud) or by `kind`, ordered chronologically.
+- `data_clean` `dryRun` parameter: no file written and no report persisted, returning the per-column cleaning plan (in `contract.columnDecisions`) plus the expected `contract`/`diffPreview`.
+- Numeric distribution `count`/`distinct` fields on numeric column profiles.
+- Config `scorecardWeights` (six per-dimension weights, non-negative, equal by default) driving a `weightedOverall` scorecard total.
+- Per-column decision trace in the clean contract (`columnDecisions`: strategy + affected rows), persisted with the clean report.
+- File-encoding reporting in `data_profile` (`encoding`: UTF-8 BOM presence and validity; invalid UTF-8 is reported, not fatal).
+- Built-in industry profile presets (`retail`, `saas`, `fund`, `real-estate`, `e-commerce`, `healthcare`, `logistics`, `manufacturing`, `energy`) via the `data_profile` `industryPreset` parameter, feeding the scorecard `accuracy` dimension; unknown ids fail loud.
+- Metric reconciliation on `data_verify`: optional `expectations` (rowCount/columnSum/columnMean/uniqueCount/nullCount, each with `column`, `expected`, optional relative `tolerance`) reconcile deterministic computed values; a mismatch is a normal `passed: false` verdict with actual/expected/tolerance detail, while invalid metrics, columns, and out-of-range tolerances fail loud. Results flow into the verify report and its persisted record.
+
+### Deviations
+
+Documented, deliberate non-goals for this repository (recorded rather than silently omitted):
+
+- **Native xlsx** — a real Excel parser needs a new dependency (or a sidecar); this repo keeps a zero-dependency parsing contract, so CSV/TSV/JSON/JSONL are the supported dataset formats, now with UTF-8/BOM encoding detection.
+- **SQL / DuckDB access** — database drivers are out of scope for the zero-dependency, in-process TypeScript seam; tabular datasets are files.
+- **`qsv` as an alternative parser** — an external binary violates the no-external-processes contract.
+- **Slot client report panel** — a browser report panel needs a client half and bundle changes; the consumption path is covered by the `data_report` tool plus storage-domain persistence instead.
+- **Computer-use fallback collection** — macOS-only and brittle; the regular filesystem capability already covers the supported collection path.
+
 ## [0.1.3] - 2026-08-23
 
 ### Changed

package/README.es.md

@@ -77,6 +77,7 @@ Todas las claves son opcionales (se muestran los valores por defecto); los valor
 | `allowedExtensions` | `['.csv', '.tsv', '.json', '.jsonl']` | Extensiones aceptadas como datasets. |
 | `workspaceRoot` | `""` | Raíz absoluta para llamadas a nivel de SERVICIO (p. ej. `verifyCitations`) sin workspace de sesión; vacío = directorio de arranque del proceso del harness. Las herramientas siempre usan el cwd del workspace de la sesión. |
 | `storeReports` | `true` | Persistir los informes en el dominio de almacenamiento `data_quality` y devolver `reportKey`. |
+| `scorecardWeights` | todo 1 (igual) | Ponderaciones por dimensión (completeness/uniqueness/validity/consistency/timeliness/accuracy) para el total ponderado del scorecard; cada peso debe ser un número no negativo. |
 
 ## Tools & surfaces
 
@@ -170,7 +171,12 @@ pnpm run verify:self-contained && pnpm run verify:artifacts && pnpm run verify:r
 
 ## Contributors
 
-Mantenido por los contribuidores de dsh-data-quality. Issues y pull requests son bienvenidos cuando el repositorio sea público.
+Gracias a todas las personas que han dado forma a este plugin.
+
+- **PerryLink** — mantenimiento y releases (`0.1.2`/`0.1.3`), mejoras de peer-dependencies, las insignias de npm version/downloads/CI, y correcciones recientes.
+- **dsh-data-quality contributors** — el scaffold inicial, el seam `ctx.dataQuality` y el contrato congelado `verifyCitations`, la capa de datasets determinista y los motores puros, los informes del dominio de almacenamiento `data_quality`, la suite vitest con servicios reales, los flujos CI/compat/release, y los READMEs en cinco idiomas.
+
+Este repositorio aún no tiene historial público de issues o pull requests; aquí se acreditarán los números de PR/issue cuando lleguen.
 
 ## PerryLink DSH Plugin Family
 

package/README.hi.md

@@ -77,6 +77,7 @@ dsh plugin --profile web remove dsh-data-quality   # अनइंस्टॉल
 | `allowedExtensions` | `['.csv', '.tsv', '.json', '.jsonl']` | डेटासेट के रूप में स्वीकृत एक्सटेंशन। |
 | `workspaceRoot` | `""` | SERVICE-स्तरीय कॉल (जैसे `verifyCitations`) के लिए निरपेक्ष रूट जिनके पास सत्र वर्कस्पेस नहीं है; रिक्त = harness प्रक्रिया का आरंभ निर्देशिका। टूल कॉल हमेशा सत्र वर्कस्पेस cwd उपयोग करती हैं। |
 | `storeReports` | `true` | रन रिपोर्ट `data_quality` स्टोरेज डोमेन में सहेजें और `reportKey` लौटाएँ। |
+| `scorecardWeights` | सभी 1 (बराबर) | स्कोरकार्ड के भारित कुल के लिए प्रति-आयाम भार (completeness/uniqueness/validity/consistency/timeliness/accuracy); हर भार एक गैर-ऋणात्मक संख्या होना चाहिए। |
 
 ## Tools & surfaces
 
@@ -170,7 +171,12 @@ pnpm run verify:self-contained && pnpm run verify:artifacts && pnpm run verify:r
 
 ## Contributors
 
-dsh-data-quality योगदानकर्ताओं द्वारा रखरखाव। रिपॉजिटरी सार्वजनिक होने पर issues और pull requests स्वागत योग्य हैं।
+इस प्लगिन को आकार देने वाले सभी का धन्यवाद।
+
+- **PerryLink** — रखरखाव और रिलीज़ (`0.1.2`/`0.1.3`), peer-dependencies उन्नयन, npm version/downloads/CI बैज, और हालिया सुधार।
+- **dsh-data-quality contributors** — प्रारंभिक scaffold, `ctx.dataQuality` सीम और फ्रोज़न `verifyCitations` अनुबंध, नियतात्मक डेटासेट परत और शुद्ध इंजन, `data_quality` स्टोरेज-डोमेन रिपोर्ट, वास्तविक-सेवा vitest सुइट, CI/compat/release वर्कफ़्लो, और पाँच-भाषा README।
+
+इस रिपॉजिटरी का अभी कोई सार्वजनिक issue या pull request इतिहास नहीं है; आने पर PR/issue संख्याएँ यहाँ अंकित की जाएँगी।
 
 ## PerryLink DSH Plugin Family
 

package/README.md

@@ -24,10 +24,11 @@ All computation is plain TypeScript in the harness process — the model never d
 ## What you get
 
 - **`ctx.dataQuality` service** — a Cordis service other plugins may optionally consume (`inject = ['dataQuality']`). Besides the three dataset operations behind the tools, it implements the frozen `verifyCitations(request)` contract: verify that numbers/strings cited in a document match a dataset snapshot, with relative-tolerance numeric comparison and `verified` / `mismatch` / `not-found` / `unverifiable` statuses.
-- **`data_profile` tool** — dataset profiling: row/column counts, inferred column types (number/date/boolean/string/empty/mixed), missing rates, unique counts, numeric distributions (min/max/mean/median/p25/p75), IQR outlier counts, mixed-type suspicion notes, and full-table duplicate-row counts. Optional deterministic systematic sampling for large files.
-- **`data_clean` tool** — ordered declarative cleaning rules: `dedupe` (by column group), `fill-missing` (constant/mean/median/forward), `coerce-type` (number/date/boolean; failures counted and set to missing), `normalize-unit` (e.g. 万/亿 suffixes to base units), `trim`, `map-values` (enum mapping). Returns a per-rule audit log plus a bounded preview; writes the cleaned dataset only when `outputPath` is given, and never overwrites the source.
+- **`data_profile` tool** — dataset profiling: row/column counts, inferred column types (number/date/boolean/string/empty/mixed), missing rates, unique counts, numeric distributions (min/max/mean/median/p25/p75), IQR outlier counts, mixed-type suspicion notes, and full-table sha256 content-hash duplicate detection with the duplicate rate and a bounded sample of duplicate row indexes. Adds a deterministic DAMA six-dimension scorecard (completeness, uniqueness, validity, consistency, timeliness, accuracy — accuracy is reported undetermined without a declared schema, never fabricated). Optional deterministic systematic sampling for large files.
+- **`data_clean` tool** — ordered declarative cleaning rules: `dedupe` (by column group), `fill-missing` (constant/mean/median/forward), `coerce-type` (number/date/boolean; failures counted and set to missing), `normalize-unit` (e.g. 万/亿 suffixes to base units), `trim`, `map-values` (enum mapping). Returns a per-rule audit log, a pre-delivery contract validation summary (dedupe before/after, uniqueness, non-null and type regressions), and a bounded preview; writes the cleaned dataset only when `outputPath` is given, and never overwrites the source.
 - **`data_verify` tool** — declarative verification rules: `not-null`, `unique`, `range`, `regex`, `enum`, `cross-column` (e.g. `startDate < endDate`), `freshness` (date column within N days of a reference date). Per-rule pass/fail with capped failing-row evidence; an overall failure is a normal `passed: false` result, not a tool error.
-- **Durable reports** — every profile/clean/verify/citation run persists to the `data_quality` storage domain (JSON backend), keyed by run timestamp plus a dataset-path fingerprint; the key is returned as `reportKey` in tool results. Clean reports also persist the bounded preview, so every model-visible result is reconstructable from its `reportKey`.
+- **`data_report` tool** — read persisted reports back from the `data_quality` storage domain: by exact `reportKey` (path-safe validation, missing keys fail loud) or by `kind` (chronological listing). Returns the report envelope(s) — kind, dataset, timestamp, and the full stored report.
+- **Durable reports** — every profile/clean/verify/citation run persists to the `data_quality` storage domain (JSON backend), keyed by run timestamp plus a dataset-path fingerprint; the key is returned as `reportKey` in tool results. Clean reports also persist the bounded preview and the contract summary, so every model-visible result is reconstructable from its `reportKey`; each clean run additionally persists a `clean-diff` before/after profile report.
 - **Session events** — on hosts that can carry them safely, runs append `data-quality/profile` / `data-quality/clean` / `data-quality/verify` events (with the `ignorable` marker where supported). On 0.1.1-rc.2 the append is skipped by design — the storage-domain report is always the durable copy (see "Known limitations").
 
 ## Quick start
@@ -83,14 +84,15 @@ All keys are optional (defaults shown); invalid values fail loudly at load. Ever
 | `allowedExtensions` | `['.csv', '.tsv', '.json', '.jsonl']` | Extensions accepted as datasets. |
 | `workspaceRoot` | `""` | Absolute root for SERVICE-level calls (e.g. `verifyCitations`) that carry no session workspace; empty = the harness process launch directory. Tool calls always use the session's workspace cwd. |
 | `storeReports` | `true` | Persist run reports to the `data_quality` storage domain and return `reportKey`. |
+| `scorecardWeights` | all 1 (equal) | Per-dimension weights (completeness/uniqueness/validity/consistency/timeliness/accuracy) for the scorecard's weighted overall total; each weight must be a non-negative number. |
 
 ## Tools & surfaces
 
-### `data_profile({ path, sample? })`
+### `data_profile({ path, sample?, industryPreset? })`
 
-Profiles a workspace dataset. `path` is workspace-relative (`.csv`/`.tsv`/`.json`/`.jsonl`; JSON must be an array of flat objects). `sample` takes every `ceil(N/sample)`-th row for the column cards (deterministic; row counts stay exact). Returns the structured report; renders a human-readable per-column summary.
+Profiles a workspace dataset. `path` is workspace-relative (`.csv`/`.tsv`/`.json`/`.jsonl`; JSON must be an array of flat objects). `sample` takes every `ceil(N/sample)`-th row for the column cards (deterministic; row counts stay exact). `industryPreset` (`retail`/`saas`/`fund`/`real-estate`/`e-commerce`/`healthcare`/`logistics`/`manufacturing`/`energy`) injects that industry's expected columns so the scorecard `accuracy` dimension becomes determinable; unknown ids fail loud. Returns the structured report — the duplicate rate, bounded duplicate-row indexes, numeric `count`/`distinct` distributions, file `encoding` (UTF-8 BOM + validity), and the weighted six-dimension DAMA scorecard — and renders a human-readable per-column summary plus scorecard.
 
-### `data_clean({ path, rules, outputPath? })`
+### `data_clean({ path, rules, outputPath?, dryRun? })`
 
 Applies `rules` in array order, each seeing the previous rule's output. Rule reference:
 
@@ -103,9 +105,13 @@ Applies `rules` in array order, each seeing the previous rule's output. Rule ref
 | `trim` | `columns?` | Trim whitespace of string cells (all columns when omitted). |
 | `map-values` | `column`, `map`, `else?` | Exact-match mapping; unmapped values stay (`keep`, default) or become `missing`. |
 
-The source file is **never** overwritten. With `outputPath` the cleaned dataset is written there (workspace-confined, format by extension); without it the run is preview-only.
+The source file is **never** overwritten. With `outputPath` the cleaned dataset is written there (workspace-confined, format by extension); without it the run is preview-only. With `dryRun: true` no file is written and nothing is persisted — the result returns the per-column cleaning plan and the expected `contract`/`diffPreview`. The result also carries a pre-delivery `contract` summary (dedupe before/after rows, uniqueness over the dedupe key or full rows, non-null regression for `fill-missing` columns, type regression for `coerce-type` columns, and the per-column decision trace), and a `clean-diff` before/after profile report is persisted to the storage domain.
 
-### `data_verify({ path, rules })`
+### `data_report({ key?, kind? })`
+
+Reads persisted reports back from the storage domain. Pass `key` (the exact `reportKey` a prior run returned) to fetch one report, or `kind` (`profile`/`clean`/`clean-diff`/`verify`/`citations`) to list every report of that kind chronologically; exactly one of `key`/`kind` is required. Malformed or missing keys fail loud.
+
+### `data_verify({ path, rules, expectations? })`
 
 Evaluates verification rules. Rule reference:
 
@@ -119,6 +125,8 @@ Evaluates verification rules. Rule reference:
 | `cross-column` | `left`, `op`, `rightColumn?`, `value?` | Compare per row: numeric when both sides parse, dates compare as epochs, strings only for `==`/`!=` (exactly one of `rightColumn`/`value`). |
 | `freshness` | `column`, `maxAgeDays`, `asOf?` | Fail dates older than `maxAgeDays` before `asOf` (default: now); unparseable/missing fails. |
 
+`expectations` reconciles deterministic metrics against expected values: `rowCount`, `columnSum`, `columnMean`, `uniqueCount`, `nullCount` (each with `column` except `rowCount`, plus `expected` and an optional relative `tolerance` in [0, 1]). Each expectation yields `passed` plus `actual`/`expected`/`tolerance`; a mismatch is a normal `passed: false` verdict, never a tool error. Invalid metrics, missing columns, and out-of-range tolerances fail loud.
+
 A missing cell fails every rule that reads it. Evidence is capped at `evidenceRowLimit` failing rows per rule.
 
 ### `ctx.dataQuality` (for other plugins)
@@ -176,7 +184,12 @@ pnpm run verify:self-contained && pnpm run verify:artifacts && pnpm run verify:r
 
 ## Contributors
 
-Maintained by the dsh-data-quality contributors. Issues and pull requests are welcome once the repository is public.
+Thanks to everyone who has shaped this plugin.
+
+- **PerryLink** — maintenance and releases (`0.1.2`/`0.1.3`), peer-dependency upgrades, the npm version/downloads/CI badges, and recent fixes.
+- **dsh-data-quality contributors** — the initial scaffold, the `ctx.dataQuality` seam and frozen `verifyCitations` contract, the deterministic dataset layer and pure engines, the `data_quality` storage-domain reports, the real-service vitest suite, the CI/compat/release workflows, and the five-language READMEs.
+
+This repository has no public issue or pull request history yet; individual PR/issue numbers will be credited here as they arrive.
 
 ## PerryLink DSH Plugin Family
 

package/README.pt.md

@@ -77,6 +77,7 @@ Todas as chaves são opcionais (valores padrão mostrados); valores inválidos f
 | `allowedExtensions` | `['.csv', '.tsv', '.json', '.jsonl']` | Extensões aceitas como datasets. |
 | `workspaceRoot` | `""` | Raiz absoluta para chamadas de nível de SERVIÇO (p. ex. `verifyCitations`) sem workspace de sessão; vazio = diretório de arranque do processo do harness. Ferramentas sempre usam o cwd do workspace da sessão. |
 | `storeReports` | `true` | Persistir relatórios no domínio de armazenamento `data_quality` e retornar `reportKey`. |
+| `scorecardWeights` | todo 1 (igual) | Pesos por dimensão (completeness/uniqueness/validity/consistency/timeliness/accuracy) para o total ponderado do scorecard; cada peso deve ser um número não negativo. |
 
 ## Tools & surfaces
 
@@ -170,7 +171,12 @@ pnpm run verify:self-contained && pnpm run verify:artifacts && pnpm run verify:r
 
 ## Contributors
 
-Mantido pelos contribuidores do dsh-data-quality. Issues e pull requests são bem-vindos quando o repositório for público.
+Obrigado a todos os que deram forma a este plugin.
+
+- **PerryLink** — manutenção e releases (`0.1.2`/`0.1.3`), atualizações de peer-dependencies, os selos npm version/downloads/CI, e correções recentes.
+- **dsh-data-quality contributors** — o scaffold inicial, o seam `ctx.dataQuality` e o contrato congelado `verifyCitations`, a camada de datasets determinística e os motores puros, os relatórios do domínio de armazenamento `data_quality`, a suíte vitest com serviços reais, os fluxos CI/compat/release, e os READMEs em cinco línguas.
+
+Este repositório ainda não tem histórico público de issues ou pull requests; os números de PR/issue serão creditados aqui quando surgirem.
 
 ## PerryLink DSH Plugin Family
 

package/README.zh.md

@@ -77,6 +77,7 @@ dsh plugin --profile web remove dsh-data-quality   # 卸载
 | `allowedExtensions` | `['.csv', '.tsv', '.json', '.jsonl']` | 可作为数据集的扩展名。 |
 | `workspaceRoot` | `""` | 服务级调用（如 `verifyCitations`）无会话工作区时使用的绝对根；空 = harness 进程启动目录。工具调用始终使用会话工作区 cwd。 |
 | `storeReports` | `true` | 把运行报告写入 `data_quality` storage domain 并返回 `reportKey`。 |
+| `scorecardWeights` | 全为 1（等权） | 六维加权总分的各维度权重（completeness/uniqueness/validity/consistency/timeliness/accuracy）；每个权重必须是非负数。 |
 
 ## Tools & surfaces
 
@@ -170,7 +171,12 @@ pnpm run verify:self-contained && pnpm run verify:artifacts && pnpm run verify:r
 
 ## Contributors
 
-由 dsh-data-quality contributors 维护。仓库公开后欢迎 issue 与 pull request。
+感谢所有塑造本插件的贡献者。
+
+- **PerryLink** — 维护与发布（`0.1.2`/`0.1.3`）、peer 依赖升级、npm 版本/下载量/CI 徽章，以及近期修复。
+- **dsh-data-quality contributors** — 初始脚手架、`ctx.dataQuality` 能力缝与冻结的 `verifyCitations` 契约、确定性数据集层与纯引擎、`data_quality` storage-domain 报告、真实服务 vitest 套件、CI/compat/release 工作流，以及五语 README。
+
+本仓库暂无公开 issue 或 pull request 历史；后续将在此逐个记名 PR/issue 编号。
 
 ## PerryLink DSH Plugin Family
 

package/cordis.patch.yml

@@ -44,3 +44,14 @@
         # Persist profile/clean/verify reports to the `data_quality` storage
         # domain (durable, queryable). Keys are returned as `reportKey`.
         storeReports: true
+        # Per-dimension weights for the scorecard's weighted overall total
+        # (completeness/uniqueness/validity/consistency/timeliness/accuracy).
+        # Each weight must be a non-negative number; omit a dimension for the
+        # equal-weight default of 1.
+        scorecardWeights:
+          completeness: 1
+          uniqueness: 1
+          validity: 1
+          consistency: 1
+          timeliness: 1
+          accuracy: 1