dsh-data-quality 0.1.0

package/README.md

@@ -0,0 +1,181 @@
+# dsh-data-quality
+
+**Deterministic data profiling, cleaning, and verification for DeepSeek Harness.**
+
+All computation is plain TypeScript in the harness process — the model never does the math. A `ctx.dataQuality` capability seam (Service Definition / local Provider / tool Consumers) exposes three model tools plus a frozen cross-plugin citation-checking contract.
+
+[English](README.md) · [简体中文](README.zh.md) · [Español](README.es.md) · [Português](README.pt.md) · [हिन्दी](README.hi.md)
+
+## Compatibility
+
+| Component | Version |
+|---|---|
+| DeepSeek Harness | `0.1.0-rc.6` (peer dependencies pinned) |
+| Node.js | `^22.19.0 \|\| >=24.0.0` |
+| Package manager | `pnpm@11.7.0` |
+| Platform | Windows / macOS / Linux (host-only plugin) |
+
+## What you get
+
+- **`ctx.dataQuality` service** — a Cordis service other plugins may optionally consume (`inject = ['dataQuality']`). Besides the three dataset operations behind the tools, it implements the frozen `verifyCitations(request)` contract: verify that numbers/strings cited in a document match a dataset snapshot, with relative-tolerance numeric comparison and `verified` / `mismatch` / `not-found` / `unverifiable` statuses.
+- **`data_profile` tool** — dataset profiling: row/column counts, inferred column types (number/date/boolean/string/empty/mixed), missing rates, unique counts, numeric distributions (min/max/mean/median/p25/p75), IQR outlier counts, mixed-type suspicion notes, and full-table duplicate-row counts. Optional deterministic systematic sampling for large files.
+- **`data_clean` tool** — ordered declarative cleaning rules: `dedupe` (by column group), `fill-missing` (constant/mean/median/forward), `coerce-type` (number/date/boolean; failures counted and set to missing), `normalize-unit` (e.g. 万/亿 suffixes to base units), `trim`, `map-values` (enum mapping). Returns a per-rule audit log plus a bounded preview; writes the cleaned dataset only when `outputPath` is given, and never overwrites the source.
+- **`data_verify` tool** — declarative verification rules: `not-null`, `unique`, `range`, `regex`, `enum`, `cross-column` (e.g. `startDate < endDate`), `freshness` (date column within N days of a reference date). Per-rule pass/fail with capped failing-row evidence; an overall failure is a normal `passed: false` result, not a tool error.
+- **Durable reports** — every profile/clean/verify/citation run persists to the `data_quality` storage domain (JSON backend), keyed by run timestamp plus a dataset-path fingerprint; the key is returned as `reportKey` in tool results.
+- **Session events** — on hosts that can carry them safely, runs append `data-quality/profile` / `data-quality/clean` / `data-quality/verify` events (with the `ignorable` marker where supported). On 0.1.0-rc.6 the append is skipped by design — the storage-domain report is always the durable copy (see "Known limitations").
+
+## Quick start
+
+### npm channel
+
+```sh
+dsh plugin --profile web add dsh-data-quality
+```
+
+### Tarball channel (no build permission needed)
+
+```sh
+pnpm pack                                  # produces dsh-data-quality-<version>.tgz
+dsh plugin --profile web add ./dsh-data-quality-<version>.tgz
+```
+
+### Git channel
+
+```sh
+dsh plugin --profile web add github:YOUR_ORG/dsh-data-quality#<commit-sha>
+```
+
+The first `add` fails because pnpm blocks the package's `prepare` build; copy the exact key pnpm printed into the profile's `pnpm-workspace.yaml` and re-run:
+
+```yaml
+allowBuilds:
+  'dsh-data-quality': true
+```
+
+Restart the profile after installing (bundles activate on restart). Then ask the agent, in a workspace containing a CSV:
+
+> Profile `holdings.csv`, then clean it by trimming whitespace, deduplicating on `fund_code`, and normalizing the `holding_value` column's 万/亿 units; finally verify `fund_code` is unique and not null.
+
+## Install & uninstall
+
+```sh
+dsh plugin --profile web add dsh-data-quality      # install (npm) — or the forms above
+dsh plugin --profile web remove dsh-data-quality   # uninstall
+```
+
+## Configuration
+
+All keys are optional (defaults shown); invalid values fail loudly at load. Every key is settable from `cordis.yml` (the bundle ships `cordis.patch.yml` with the same defaults).
+
+| Key | Default | Description |
+|---|---|---|
+| `enabled` | `true` | Master switch; `false` mounts nothing at all. |
+| `maxRows` | `200000` | Hard row cap per dataset load; larger inputs reject loudly (use the tool's `sample` parameter). |
+| `maxFileSizeMB` | `64` | Hard file-size cap in MiB per dataset load. |
+| `defaultTolerance` | `1e-9` | Default relative tolerance for numeric citation comparison when a citation omits `tolerance`. |
+| `evidenceRowLimit` | `20` | Cap on failing-row evidence (verify) and preview rows (clean) in one result. |
+| `allowedExtensions` | `['.csv', '.tsv', '.json', '.jsonl']` | Extensions accepted as datasets. |
+| `workspaceRoot` | `""` | Absolute root for SERVICE-level calls (e.g. `verifyCitations`) that carry no session workspace; empty = the harness process launch directory. Tool calls always use the session's workspace cwd. |
+| `storeReports` | `true` | Persist run reports to the `data_quality` storage domain and return `reportKey`. |
+
+## Tools & surfaces
+
+### `data_profile({ path, sample? })`
+
+Profiles a workspace dataset. `path` is workspace-relative (`.csv`/`.tsv`/`.json`/`.jsonl`; JSON must be an array of flat objects). `sample` takes every `ceil(N/sample)`-th row for the column cards (deterministic; row counts stay exact). Returns the structured report; renders a human-readable per-column summary.
+
+### `data_clean({ path, rules, outputPath? })`
+
+Applies `rules` in array order, each seeing the previous rule's output. Rule reference:
+
+| Rule | Extra fields | Semantics |
+|---|---|---|
+| `dedupe` | `columns?` | Remove rows whose key-column values duplicate an earlier row (first kept; all columns when omitted). |
+| `fill-missing` | `column`, `strategy`, `value?` | Fill missing cells: `constant` (needs `value`), `mean`/`median` (numeric columns), `forward` (previous non-missing). |
+| `coerce-type` | `column`, `to` | Coerce to `number`/`date` (ISO)/`boolean`; failures become missing and are counted in the log. |
+| `normalize-unit` | `column`, `factors` | Strip a unit suffix and multiply (`{"万": 10000, "亿": 100000000}`); plain numerics convert too. |
+| `trim` | `columns?` | Trim whitespace of string cells (all columns when omitted). |
+| `map-values` | `column`, `map`, `else?` | Exact-match mapping; unmapped values stay (`keep`, default) or become `missing`. |
+
+The source file is **never** overwritten. With `outputPath` the cleaned dataset is written there (workspace-confined, format by extension); without it the run is preview-only.
+
+### `data_verify({ path, rules })`
+
+Evaluates verification rules. Rule reference:
+
+| Rule | Extra fields | Semantics |
+|---|---|---|
+| `not-null` | `column` | Fail missing cells (null/empty/whitespace). |
+| `unique` | `columns` | Fail every row whose key combination repeats (missing participates). |
+| `range` | `column`, `min?`, `max?` | Fail missing/unparseable cells and values outside the inclusive bounds (at least one bound required). |
+| `regex` | `column`, `pattern`, `flags?` | Fail missing or non-matching cells (full JS regex). |
+| `enum` | `column`, `values` | Fail cells whose trimmed text is not listed. |
+| `cross-column` | `left`, `op`, `rightColumn?`, `value?` | Compare per row: numeric when both sides parse, dates compare as epochs, strings only for `==`/`!=` (exactly one of `rightColumn`/`value`). |
+| `freshness` | `column`, `maxAgeDays`, `asOf?` | Fail dates older than `maxAgeDays` before `asOf` (default: now); unparseable/missing fails. |
+
+A missing cell fails every rule that reads it. Evidence is capped at `evidenceRowLimit` failing rows per rule.
+
+### `ctx.dataQuality` (for other plugins)
+
+```ts
+const result = await ctx.dataQuality.verifyCitations({
+  dataset: 'holdings.csv',          // resolved against workspaceRoot
+  citations: [
+    { id: 'c1', path: 'rows[3].nav', value: 1.234, tolerance: 0.01 },
+    { id: 'c2', path: 'summary.annualReturn', value: '12.34%' },
+  ],
+})
+// result.results[i] = { id, status: 'verified' | 'mismatch' | 'not-found' | 'unverifiable', actual?, note? }
+```
+
+Locators walk the dataset document: CSV/TSV load as `{ columns, rows }` (so `rows[3].nav` resolves), JSON is the parsed value, JSONL the array of parsed lines. Numbers compare with relative tolerance (`|a-b| <= tolerance * max(|a|, |b|)`); a CSV string cell that parses numerically compares as a number; strings compare exactly; incomparable type pairs are `unverifiable`. The service also exposes `profileDataset` / `cleanDataset` / `verifyDataset` (the same operations the tools call).
+
+## Permissions & data
+
+- **Reads** workspace dataset files (allowlisted extensions only).
+- **Writes** only: the `data_clean` output file (explicit `outputPath`, workspace-confined, never the input) and reports in the `data_quality` storage domain under the harness data directory.
+- **No network, no credentials, no external processes** — all parsing and statistics are in-process TypeScript.
+- Reports may contain sample cell values from your datasets (bounded by `evidenceRowLimit` and display truncation); the session log records tool arguments and results as usual.
+
+## Security boundaries
+
+- **Path confinement** — dataset and output paths must resolve inside the session workspace (`verifyCitations` uses `workspaceRoot`); `..` escapes and outside absolute paths reject, and both sides are normalized before comparison (Windows slash-safe).
+- **Bounded work** — `maxRows` / `maxFileSizeMB` guards reject oversized inputs loudly; abort signals cancel long loads mid-stream.
+- **No overwrite** — `data_clean` refuses an `outputPath` equal to the input path.
+- **Deterministic computation** — same input, same output; the only clock is the one injected for `freshness` defaults and report timestamps.
+
+## Known limitations
+
+- **Session events are adaptive.** 0.1.0-rc.6 has no plugin session-event registration surface and its `Session.append` cannot stamp the `ignorable` marker, so appending an unknown `data-quality/*` type would make the session log unreadable on restore. The plugin therefore appends only when the host knows the vocabulary or supports the `ignorable` append flag; on rc.6 the storage-domain report is the durable record.
+- **CSV dialect** — comma/tab with RFC-4180 quoting, header row required, blank lines skipped, no delimiter auto-detection or comment lines.
+- **Type parsing is strict** — numbers have no thousands separators; dates are `YYYY-MM-DD` / `YYYY/MM/DD` / ISO-like datetimes (UTC); booleans are `true/false/yes/no/1/0`. Everything else profiles as `string`/`mixed` — clean it with `coerce-type` when intended.
+- **JSON must be tabular for the tools** (array of flat objects); `verifyCitations` walks arbitrary JSON documents.
+- **No ML anomaly detection, no PII masking, no databases, no SQL** — rule-based suspicion notes only.
+
+## Development
+
+```sh
+pnpm install
+pnpm run typecheck && pnpm run typecheck:ci && pnpm test && pnpm run build
+pnpm run verify:self-contained && pnpm run verify:artifacts && pnpm run verify:readme-sync && pnpm pack
+```
+
+- Tests run vitest against the REAL `Context`/`Session`/`ToolRuntime`/storage domain from the 0.1.0-rc.6 peers (no hand-written service mocks) plus pure engine specs; every clean/verify rule has positive and negative cases, and `verifyCitations` covers all four statuses.
+- `scripts/loader-runner.mjs` boots the real Loader composition and executes the profile → clean → verify chain against `fixtures/` without an API key.
+- Release: `node scripts/release.mjs <x.y.z>` (never pushes; the tag triggers `release.yml`).
+
+## Topics
+
+`dsh` · `dsh-plugin` · `deepseek-harness` · `cordis` · `data-quality` · `data-cleaning` · `data-profiling` · `data-verification`
+
+## Contributors
+
+Maintained by the dsh-data-quality contributors. Issues and pull requests are welcome once the repository is public.
+
+## PerryLink DSH Plugin Family
+
+This plugin follows the shared DSH family engineering conventions: bundle-manifest packaging (`dsh.bundle` + `cordis.patch.yml`), five-language READMEs gated by a sync check, fail-loud Schemastery configuration, real-service vitest coverage, and a three-workflow CI/compat/release chain.
+
+## License
+
+Apache-2.0 — see [LICENSE](LICENSE) and [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md).

package/README.pt.md

@@ -0,0 +1,181 @@
+# dsh-data-quality
+
+**Perfilamento, limpeza e verificação de dados determinísticos para DeepSeek Harness.**
+
+Todo o cálculo é TypeScript puro no processo do harness — o modelo nunca faz as contas. Uma costura de capacidade `ctx.dataQuality` (Service Definition / Provider local / Consumers de ferramentas) expõe três ferramentas para o modelo mais um contrato congelado de verificação de citações entre plugins.
+
+[English](README.md) · [简体中文](README.zh.md) · [Español](README.es.md) · [Português](README.pt.md) · [हिन्दी](README.hi.md)
+
+## Compatibility
+
+| Componente | Versão |
+|---|---|
+| DeepSeek Harness | `0.1.0-rc.6` (dependências peer fixadas) |
+| Node.js | `^22.19.0 \|\| >=24.0.0` |
+| Gerenciador de pacotes | `pnpm@11.7.0` |
+| Plataforma | Windows / macOS / Linux (plugin apenas de host) |
+
+## What you get
+
+- **Serviço `ctx.dataQuality`** — um serviço Cordis que outros plugins podem consumir opcionalmente (`inject = ['dataQuality']`). Além das três operações sobre datasets por trás das ferramentas, implementa o contrato congelado `verifyCitations(request)`: verifica se números/strings citados num documento batem com um instantâneo do dataset, com comparação numérica por tolerância relativa e estados `verified` / `mismatch` / `not-found` / `unverifiable`.
+- **Ferramenta `data_profile`** — perfilamento de datasets: contagens de linhas/colunas, tipos de coluna inferidos (number/date/boolean/string/empty/mixed), taxas de ausência, contagens de valores únicos, distribuições numéricas (min/max/mean/median/p25/p75), contagem de outliers IQR, notas de suspeita de tipos mistos e contagem de linhas duplicadas da tabela inteira. Amostragem sistemática determinística opcional para arquivos grandes.
+- **Ferramenta `data_clean`** — regras declarativas de limpeza em ordem: `dedupe` (por grupo de colunas), `fill-missing` (constant/mean/median/forward), `coerce-type` (number/date/boolean; falhas contadas e viram ausentes), `normalize-unit` (p. ex. sufixos 万/亿 para unidades base), `trim`, `map-values` (mapeamento de enumerações). Retorna um log de auditoria por regra mais uma prévia limitada; só grava o dataset limpo quando `outputPath` é dado e nunca sobrescreve a origem.
+- **Ferramenta `data_verify`** — regras declarativas de verificação: `not-null`, `unique`, `range`, `regex`, `enum`, `cross-column` (p. ex. `startDate < endDate`), `freshness` (coluna de data dentro de N dias de uma data de referência). pass/fail por regra com evidência limitada de linhas falhas; uma falha geral é um resultado normal `passed: false`, não um erro de ferramenta.
+- **Relatórios duráveis** — cada execução de perfilamento/limpeza/verificação/citações persiste no domínio de armazenamento `data_quality` (backend JSON), com chave de timestamp mais impressão digital do caminho do dataset; a chave é retornada como `reportKey` nos resultados.
+- **Eventos de sessão** — em hosts que os suportam com segurança, as execuções anexam eventos `data-quality/profile` / `data-quality/clean` / `data-quality/verify` (com a marca `ignorable` onde suportado). Em 0.1.0-rc.6 o append é omitido por design — o relatório do domínio de armazenamento é sempre a cópia durável (ver «Known limitations»).
+
+## Quick start
+
+### Canal npm
+
+```sh
+dsh plugin --profile web add dsh-data-quality
+```
+
+### Canal tarball (não precisa de permissão de build)
+
+```sh
+pnpm pack                                  # produz dsh-data-quality-<version>.tgz
+dsh plugin --profile web add ./dsh-data-quality-<version>.tgz
+```
+
+### Canal git
+
+```sh
+dsh plugin --profile web add github:YOUR_ORG/dsh-data-quality#<commit-sha>
+```
+
+O primeiro `add` falha porque o pnpm bloqueia o build `prepare` do pacote; copie a chave exata que o pnpm imprimiu para o `pnpm-workspace.yaml` do profile e execute de novo:
+
+```yaml
+allowBuilds:
+  'dsh-data-quality': true
+```
+
+Reinicie o profile após instalar (bundles ativam no reinício). Depois peça ao agente, num workspace com um CSV:
+
+> Perfile `holdings.csv`, depois limpe-o aparando espaços, desduplicando por `fund_code` e normalizando as unidades 万/亿 da coluna `holding_value`; por fim verifique que `fund_code` é único e não nulo.
+
+## Install & uninstall
+
+```sh
+dsh plugin --profile web add dsh-data-quality      # instalar (npm) — ou as formas acima
+dsh plugin --profile web remove dsh-data-quality   # desinstalar
+```
+
+## Configuration
+
+Todas as chaves são opcionais (valores padrão mostrados); valores inválidos falham ruidosamente no carregamento. Cada chave pode ser alterada no `cordis.yml` (o bundle inclui `cordis.patch.yml` com os mesmos padrões).
+
+| Key | Default | Description |
+|---|---|---|
+| `enabled` | `true` | Interruptor mestre; `false` não monta nada. |
+| `maxRows` | `200000` | Teto rígido de linhas por carga; entradas maiores são rejeitadas ruidosamente (use o parâmetro `sample` da ferramenta). |
+| `maxFileSizeMB` | `64` | Teto rígido de tamanho de arquivo em MiB por carga. |
+| `defaultTolerance` | `1e-9` | Tolerância relativa padrão para comparação numérica de citações quando a citação omite `tolerance`. |
+| `evidenceRowLimit` | `20` | Teto de linhas de evidência (verify) e de prévia (clean) num resultado. |
+| `allowedExtensions` | `['.csv', '.tsv', '.json', '.jsonl']` | Extensões aceitas como datasets. |
+| `workspaceRoot` | `""` | Raiz absoluta para chamadas de nível de SERVIÇO (p. ex. `verifyCitations`) sem workspace de sessão; vazio = diretório de arranque do processo do harness. Ferramentas sempre usam o cwd do workspace da sessão. |
+| `storeReports` | `true` | Persistir relatórios no domínio de armazenamento `data_quality` e retornar `reportKey`. |
+
+## Tools & surfaces
+
+### `data_profile({ path, sample? })`
+
+Perfilam um dataset do workspace. `path` é relativo ao workspace (`.csv`/`.tsv`/`.json`/`.jsonl`; JSON deve ser um array de objetos planos). `sample` toma cada `ceil(N/sample)`-ésima linha para os cartões de coluna (determinístico; as contagens de linhas continuam exatas). Retorna o relatório estruturado e renderiza um resumo legível por coluna.
+
+### `data_clean({ path, rules, outputPath? })`
+
+Aplica `rules` na ordem do array; cada regra vê a saída da anterior. Referência de regras:
+
+| Regra | Campos extras | Semântica |
+|---|---|---|
+| `dedupe` | `columns?` | Remove linhas cuja combinação de colunas-chave duplica uma linha anterior (a primeira é mantida; todas as colunas se omitido). |
+| `fill-missing` | `column`, `strategy`, `value?` | Preenche ausentes: `constant` (requer `value`), `mean`/`median` (colunas numéricas), `forward` (valor anterior não ausente). |
+| `coerce-type` | `column`, `to` | Converte para `number`/`date` (ISO)/`boolean`; falhas viram ausentes e são contadas. |
+| `normalize-unit` | `column`, `factors` | Remove o sufixo de unidade e multiplica (`{"万": 10000, "亿": 100000000}`); numéricos simples também convertem. |
+| `trim` | `columns?` | Apara espaços de células de texto (todas as colunas se omitido). |
+| `map-values` | `column`, `map`, `else?` | Mapeamento por correspondência exata; valores não mapeados ficam (`keep`, padrão) ou viram `missing`. |
+
+O arquivo de origem **nunca** é sobrescrito. Com `outputPath` o dataset limpo é gravado lá (confinado ao workspace, formato por extensão); sem ele a execução é apenas prévia.
+
+### `data_verify({ path, rules })`
+
+Avalia regras de verificação. Referência de regras:
+
+| Regra | Campos extras | Semântica |
+|---|---|---|
+| `not-null` | `column` | Falham células ausentes (null/vazio/só espaços). |
+| `unique` | `columns` | Falha cada linha cuja combinação-chave se repete (ausentes participam). |
+| `range` | `column`, `min?`, `max?` | Falham células ausentes/não parseáveis e valores fora dos limites inclusivos (pelo menos um limite obrigatório). |
+| `regex` | `column`, `pattern`, `flags?` | Falham células ausentes ou não correspondentes (regex JS completa). |
+| `enum` | `column`, `values` | Falham células cujo texto aparado não está na lista. |
+| `cross-column` | `left`, `op`, `rightColumn?`, `value?` | Compara por linha: numérico quando ambos os lados parseiam, datas como épocas, strings só para `==`/`!=` (exatamente um de `rightColumn`/`value`). |
+| `freshness` | `column`, `maxAgeDays`, `asOf?` | Falham datas mais velhas que `maxAgeDays` antes de `asOf` (padrão: agora); não parseável/ausente falha. |
+
+Uma célula ausente faz falhar toda regra que a lê. A evidência é limitada a `evidenceRowLimit` linhas falhas por regra.
+
+### `ctx.dataQuality` (para outros plugins)
+
+```ts
+const result = await ctx.dataQuality.verifyCitations({
+  dataset: 'holdings.csv',          // resolvido contra workspaceRoot
+  citations: [
+    { id: 'c1', path: 'rows[3].nav', value: 1.234, tolerance: 0.01 },
+    { id: 'c2', path: 'summary.annualReturn', value: '12.34%' },
+  ],
+})
+// result.results[i] = { id, status: 'verified' | 'mismatch' | 'not-found' | 'unverifiable', actual?, note? }
+```
+
+Os localizadores percorrem o documento do dataset: CSV/TSV carregam como `{ columns, rows }` (logo `rows[3].nav` resolve), JSON é o valor parseado, JSONL o array de linhas parseadas. Números comparam com tolerância relativa (`|a-b| <= tolerance * max(|a|, |b|)`); uma célula de texto CSV que parseia numericamente compara como número; strings comparam exatas; pares de tipos incomparáveis são `unverifiable`. O serviço também expõe `profileDataset` / `cleanDataset` / `verifyDataset` (as mesmas operações que as ferramentas chamam).
+
+## Permissions & data
+
+- **Lê** arquivos de dataset do workspace (apenas extensões permitidas).
+- **Escreve** apenas: o arquivo de saída do `data_clean` (`outputPath` explícito, confinado ao workspace, nunca a entrada) e os relatórios do domínio de armazenamento `data_quality` no diretório de dados do harness.
+- **Sem rede, sem credenciais, sem processos externos** — todo o parsing e a estatística são TypeScript em processo.
+- Os relatórios podem conter valores de célula de amostra dos seus datasets (limitados por `evidenceRowLimit` e pelo truncamento de exibição); o log de sessão regista argumentos e resultados de ferramentas como de costume.
+
+## Security boundaries
+
+- **Confinamento de caminhos** — caminhos de dataset e saída devem resolver dentro do workspace de sessão (`verifyCitations` usa `workspaceRoot`); escapes `..` e caminhos absolutos fora da raiz são rejeitados, e ambos os lados são normalizados antes da comparação (seguro com barras do Windows).
+- **Trabalho limitado** — as guardas `maxRows` / `maxFileSizeMB` rejeitam entradas sobredimensionadas ruidosamente; sinais de aborto cancelam cargas longas no meio.
+- **Sem sobrescrita** — `data_clean` recusa um `outputPath` igual ao caminho de entrada.
+- **Cálculo determinístico** — mesma entrada, mesma saída; o único relógio é o injetado para os padrões de `freshness` e os timestamps de relatórios.
+
+## Known limitations
+
+- **Os eventos de sessão são adaptativos.** 0.1.0-rc.6 não tem superfície de registo de eventos de sessão para plugins e o seu `Session.append` não consegue estampar a marca `ignorable`, logo anexar um tipo `data-quality/*` desconhecido tornaria o log de sessão ilegível ao restaurar. Por isso o plugin só anexa quando o host conhece o vocabulário ou suporta o flag `ignorable`; em rc.6 o relatório do domínio de armazenamento é o registo durável.
+- **Dialeto CSV** — vírgula/tab com aspas RFC-4180, linha de cabeçalho obrigatória, linhas em branco ignoradas; sem autodetecção de delimitador nem linhas de comentário.
+- **O parsing de tipos é estrito** — números sem separadores de milhares; datas são `YYYY-MM-DD` / `YYYY/MM/DD` / datetimes estilo ISO (UTC); booleanos são `true/false/yes/no/1/0`. Todo o resto é perfilado como `string`/`mixed` — limpe com `coerce-type` se for intencional.
+- **JSON tem de ser tabular para as ferramentas** (array de objetos planos); `verifyCitations` percorre documentos JSON arbitrários.
+- **Sem deteção de anomalias por ML, sem mascaramento de PII, sem bases de dados, sem SQL** — apenas notas de suspeita baseadas em regras.
+
+## Development
+
+```sh
+pnpm install
+pnpm run typecheck && pnpm run typecheck:ci && pnpm test && pnpm run build
+pnpm run verify:self-contained && pnpm run verify:artifacts && pnpm run verify:readme-sync && pnpm pack
+```
+
+- Os testes correm vitest contra os `Context`/`Session`/`ToolRuntime`/domínio de armazenamento REAIS dos peers 0.1.0-rc.6 (sem mocks de serviços escritos à mão) mais specs de motores puros; cada regra de limpeza/verificação tem casos positivos e negativos, e `verifyCitations` cobre os quatro estados.
+- `scripts/loader-runner.mjs` arranca a composição real do Loader e executa a cadeia perfilar → limpar → verificar contra `fixtures/` sem chave de API.
+- Release: `node scripts/release.mjs <x.y.z>` (nunca faz push; a tag dispara `release.yml`).
+
+## Topics
+
+`dsh` · `dsh-plugin` · `deepseek-harness` · `cordis` · `data-quality` · `data-cleaning` · `data-profiling` · `data-verification`
+
+## Contributors
+
+Mantido pelos contribuidores do dsh-data-quality. Issues e pull requests são bem-vindos quando o repositório for público.
+
+## PerryLink DSH Plugin Family
+
+Este plugin segue as convenções de engenharia partilhadas da família DSH: empacotamento com manifesto bundle (`dsh.bundle` + `cordis.patch.yml`), READMEs em cinco línguas com verificação de sincronia, configuração Schemastery de falha ruidosa, cobertura vitest com serviços reais e a cadeia de três fluxos CI/compat/release.
+
+## License
+
+Apache-2.0 — ver [LICENSE](LICENSE) e [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md).

package/README.zh.md

@@ -0,0 +1,181 @@
+# dsh-data-quality
+
+**DeepSeek Harness 的确定性数据梳理、清洗与核查插件。**
+
+全部计算为 harness 进程内的纯 TypeScript —— 模型从不心算。`ctx.dataQuality` 能力缝隙（Service Definition / 本地 Provider / 工具 Consumer）对外提供三个模型工具，以及一份冻结的跨插件引用核查契约。
+
+[English](README.md) · [简体中文](README.zh.md) · [Español](README.es.md) · [Português](README.pt.md) · [हिन्दी](README.hi.md)
+
+## Compatibility
+
+| 组件 | 版本 |
+|---|---|
+| DeepSeek Harness | `0.1.0-rc.6`（peer 依赖钉版） |
+| Node.js | `^22.19.0 \|\| >=24.0.0` |
+| 包管理器 | `pnpm@11.7.0` |
+| 平台 | Windows / macOS / Linux（纯宿主插件） |
+
+## What you get
+
+- **`ctx.dataQuality` 服务** —— 其他插件可选消费的 Cordis 服务（`inject = ['dataQuality']`）。除支撑三个工具的数据集操作外，还实现冻结的 `verifyCitations(request)` 契约：核查文档中引用的数字/字符串是否与数据集快照一致，数值按相对容差比较，状态为 `verified` / `mismatch` / `not-found` / `unverifiable`。
+- **`data_profile` 工具** —— 数据集梳理：行/列数、推断列类型（number/date/boolean/string/empty/mixed）、缺失率、唯一值数、数值分布（min/max/mean/median/p25/p75）、IQR 离群计数、混合类型疑点提示、全表重复行计数。大文件可选确定性系统采样。
+- **`data_clean` 工具** —— 有序声明式清洗规则：`dedupe`（按列组）、`fill-missing`（常量/均值/中位数/前向填充）、`coerce-type`（number/date/boolean，失败计数并置缺失）、`normalize-unit`（万/亿 等单位后缀归一）、`trim`、`map-values`（枚举映射）。返回逐规则审计日志与有界预览；仅在给出 `outputPath` 时落盘，且绝不覆盖源文件。
+- **`data_verify` 工具** —— 声明式核查规则：`not-null`、`unique`、`range`、`regex`、`enum`、`cross-column`（如 `startDate < endDate`）、`freshness`（日期列距参考日期不超过 N 天）。逐规则 pass/fail 并附有界失败行证据；整体失败是正常结果 `passed: false`，不是工具错误。
+- **持久化报告** —— 每次梳理/清洗/核查/引用检查都写入 `data_quality` storage domain（JSON 后端），键为运行时间戳加数据集路径指纹；工具结果以 `reportKey` 返回该键。
+- **会话事件** —— 宿主支持时，运行会追加 `data-quality/profile` / `data-quality/clean` / `data-quality/verify` 事件（支持处带 `ignorable` 标记）。在 0.1.0-rc.6 上按设计跳过 append —— storage domain 报告始终是持久副本（见「Known limitations」）。
+
+## Quick start
+
+### npm 通道
+
+```sh
+dsh plugin --profile web add dsh-data-quality
+```
+
+### Tarball 通道（无需构建授权）
+
+```sh
+pnpm pack                                  # 产出 dsh-data-quality-<version>.tgz
+dsh plugin --profile web add ./dsh-data-quality-<version>.tgz
+```
+
+### Git 通道
+
+```sh
+dsh plugin --profile web add github:YOUR_ORG/dsh-data-quality#<commit-sha>
+```
+
+第一次 `add` 会失败，因为 pnpm 默认阻止该包的 `prepare` 构建；把 pnpm 打印的键复制到 profile 的 `pnpm-workspace.yaml` 后重试：
+
+```yaml
+allowBuilds:
+  'dsh-data-quality': true
+```
+
+安装后重启 profile 生效（bundle 重启激活）。然后在包含 CSV 的工作区中对 agent 说：
+
+> 先梳理 `holdings.csv`，再按规则清洗：去空白、按 `fund_code` 去重、把 `holding_value` 列的 万/亿 单位归一；最后核查 `fund_code` 唯一且非空。
+
+## Install & uninstall
+
+```sh
+dsh plugin --profile web add dsh-data-quality      # 安装（npm）——或用上述其他形式
+dsh plugin --profile web remove dsh-data-quality   # 卸载
+```
+
+## Configuration
+
+所有键均可选（所示为默认值）；非法值在加载期响亮失败。每个键都可从 `cordis.yml` 修改（bundle 自带的 `cordis.patch.yml` 写有相同默认值）。
+
+| Key | Default | Description |
+|---|---|---|
+| `enabled` | `true` | 总开关；`false` 时完全不挂载。 |
+| `maxRows` | `200000` | 单次加载的硬行数上限；超限响亮拒绝（可用工具的 `sample` 参数）。 |
+| `maxFileSizeMB` | `64` | 单次加载的硬文件大小上限（MiB）。 |
+| `defaultTolerance` | `1e-9` | 引用未给 `tolerance` 时数值比较的默认相对容差。 |
+| `evidenceRowLimit` | `20` | 单结果中失败行证据（核查）与预览行（清洗）的上限。 |
+| `allowedExtensions` | `['.csv', '.tsv', '.json', '.jsonl']` | 可作为数据集的扩展名。 |
+| `workspaceRoot` | `""` | 服务级调用（如 `verifyCitations`）无会话工作区时使用的绝对根；空 = harness 进程启动目录。工具调用始终使用会话工作区 cwd。 |
+| `storeReports` | `true` | 把运行报告写入 `data_quality` storage domain 并返回 `reportKey`。 |
+
+## Tools & surfaces
+
+### `data_profile({ path, sample? })`
+
+梳理工作区数据集。`path` 为工作区相对路径（`.csv`/`.tsv`/`.json`/`.jsonl`；JSON 必须是扁平对象数组）。`sample` 按每 `ceil(N/sample)` 行取样计算列卡片（确定性；行数仍精确）。返回结构化报告，并渲染人类可读的逐列摘要。
+
+### `data_clean({ path, rules, outputPath? })`
+
+按数组顺序应用 `rules`，每条规则看到上一条的输出。规则参考：
+
+| 规则 | 额外字段 | 语义 |
+|---|---|---|
+| `dedupe` | `columns?` | 删除键列组合与前行重复的行（保留首次；省略时为全部列）。 |
+| `fill-missing` | `column`, `strategy`, `value?` | 填充缺失：`constant`（需 `value`）、`mean`/`median`（数值列）、`forward`（前一个非缺失值）。 |
+| `coerce-type` | `column`, `to` | 转换为 `number`/`date`（ISO）/`boolean`；失败置缺失并计数入日志。 |
+| `normalize-unit` | `column`, `factors` | 剥离单位后缀并乘系数（`{"万": 10000, "亿": 100000000}`）；纯数值也转换。 |
+| `trim` | `columns?` | 去除字符串单元格首尾空白（省略时为全部列）。 |
+| `map-values` | `column`, `map`, `else?` | 精确匹配映射；未映射值保留（`keep`，默认）或置缺失（`missing`）。 |
+
+源文件**绝不**被覆盖。给出 `outputPath` 时清洗结果写入该路径（限定工作区内，按扩展名定格式）；否则仅预览。
+
+### `data_verify({ path, rules })`
+
+评估核查规则。规则参考：
+
+| 规则 | 额外字段 | 语义 |
+|---|---|---|
+| `not-null` | `column` | 缺失（null/空/纯空白）即失败。 |
+| `unique` | `columns` | 键组合重复的每一行都失败（缺失值参与判重）。 |
+| `range` | `column`, `min?`, `max?` | 缺失/不可解析或超出闭区间即失败（至少需一个界）。 |
+| `regex` | `column`, `pattern`, `flags?` | 缺失或不匹配即失败（完整 JS 正则）。 |
+| `enum` | `column`, `values` | 去空白后的文本不在枚举内即失败。 |
+| `cross-column` | `left`, `op`, `rightColumn?`, `value?` | 逐行比较：两侧可数值解析按数值，可日期解析按纪元毫秒，否则字符串仅支持 `==`/`!=`（`rightColumn`/`value` 恰给一个）。 |
+| `freshness` | `column`, `maxAgeDays`, `asOf?` | 日期早于 `asOf` 前 `maxAgeDays` 天即失败（`asOf` 默认当前）；不可解析/缺失即失败。 |
+
+任何被读取的单元格缺失都会使该规则该行失败。每条规则的失败行证据上限为 `evidenceRowLimit`。
+
+### `ctx.dataQuality`（供其他插件）
+
+```ts
+const result = await ctx.dataQuality.verifyCitations({
+  dataset: 'holdings.csv',          // 相对 workspaceRoot 解析
+  citations: [
+    { id: 'c1', path: 'rows[3].nav', value: 1.234, tolerance: 0.01 },
+    { id: 'c2', path: 'summary.annualReturn', value: '12.34%' },
+  ],
+})
+// result.results[i] = { id, status: 'verified' | 'mismatch' | 'not-found' | 'unverifiable', actual?, note? }
+```
+
+定位符在数据集文档上行走：CSV/TSV 加载为 `{ columns, rows }`（故 `rows[3].nav` 可解析），JSON 为解析值本身，JSONL 为逐行解析值数组。数值按相对容差比较（`|a-b| <= tolerance * max(|a|, |b|)`）；可数值解析的 CSV 字符串单元格按数值比较；字符串精确比较；类型不可比为 `unverifiable`。服务另暴露 `profileDataset` / `cleanDataset` / `verifyDataset`（即三个工具调用的同一实现）。
+
+## Permissions & data
+
+- **读取**工作区数据集文件（仅白名单扩展名）。
+- **写入**仅有：`data_clean` 的输出文件（显式 `outputPath`、限定工作区内、绝不覆盖输入）与 harness 数据目录下 `data_quality` storage domain 中的报告。
+- **无网络、无凭据、无外部进程** —— 全部解析与统计都是进程内 TypeScript。
+- 报告可能包含数据集的样本单元格值（受 `evidenceRowLimit` 与展示截断约束）；会话日志照常记录工具参数与结果。
+
+## Security boundaries
+
+- **路径限定** —— 数据集与输出路径必须解析在会话工作区内（`verifyCitations` 用 `workspaceRoot`）；拒绝 `..` 逃逸与根外绝对路径，比较前双侧归一化（Windows 斜杠安全）。
+- **有界工作量** —— `maxRows` / `maxFileSizeMB` 守卫对超限输入响亮拒绝；abort 信号可中断长加载。
+- **不覆盖** —— `data_clean` 拒绝与输入相同的 `outputPath`。
+- **确定性计算** —— 相同输入相同输出；唯一的时钟是为 `freshness` 默认值与报告时间戳注入的时钟。
+
+## Known limitations
+
+- **会话事件是自适应的。** 0.1.0-rc.6 没有插件会话事件注册面，`Session.append` 也无法打 `ignorable` 标记；追加未知 `data-quality/*` 类型会让会话日志在恢复时被拒读。因此插件仅在宿主认识该词汇或支持 `ignorable` append 时才追加；在 rc.6 上 storage domain 报告即持久记录。
+- **CSV 方言** —— 逗号/制表符分隔、RFC-4180 引号、首行必须是表头、跳过空白行；无分隔符自动探测、无注释行。
+- **类型解析是严格的** —— 数字不带千分位；日期为 `YYYY-MM-DD` / `YYYY/MM/DD` / ISO 风格时间（UTC）；布尔为 `true/false/yes/no/1/0`。其余一律按 `string`/`mixed` 梳理 —— 如有意图请用 `coerce-type` 清洗。
+- **JSON 对工具必须是表格**（扁平对象数组）；`verifyCitations` 可行走任意 JSON 文档。
+- **不做 ML 异常检测、不做 PII 脱敏、不连数据库、不做 SQL** —— 仅规则式疑点提示。
+
+## Development
+
+```sh
+pnpm install
+pnpm run typecheck && pnpm run typecheck:ci && pnpm test && pnpm run build
+pnpm run verify:self-contained && pnpm run verify:artifacts && pnpm run verify:readme-sync && pnpm pack
+```
+
+- 测试用 vitest 跑 0.1.0-rc.6 peers 的真实 `Context`/`Session`/`ToolRuntime`/storage domain（不手写服务 mock）加纯引擎用例；每类清洗/核查规则都有正反用例，`verifyCitations` 覆盖四种状态。
+- `scripts/loader-runner.mjs` 以真实 Loader 组合启动，并在无 API key 下对 `fixtures/` 执行 梳理 → 清洗 → 核查 链路。
+- 发布：`node scripts/release.mjs <x.y.z>`（绝不 push；tag 触发 `release.yml`）。
+
+## Topics
+
+`dsh` · `dsh-plugin` · `deepseek-harness` · `cordis` · `data-quality` · `data-cleaning` · `data-profiling` · `data-verification`
+
+## Contributors
+
+由 dsh-data-quality contributors 维护。仓库公开后欢迎 issue 与 pull request。
+
+## PerryLink DSH Plugin Family
+
+本插件遵循 DSH 家族共享工程规范：bundle 清单打包（`dsh.bundle` + `cordis.patch.yml`）、同步门禁约束的五语 README、响亮失败的 Schemastery 配置、真实服务 vitest 覆盖、CI/compat/release 三工作流链。
+
+## License
+
+Apache-2.0 —— 见 [LICENSE](LICENSE) 与 [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md)。

package/THIRD_PARTY_NOTICES.md

@@ -0,0 +1,20 @@
+# Third-party notices
+
+`dsh-data-quality` bundles no third-party source code. All TypeScript/JavaScript
+sources in this repository are original works by the dsh-data-quality contributors,
+licensed under Apache-2.0 (see `LICENSE`).
+
+The package depends on the following software. None of it is bundled into the
+published tarball except where noted; these are install-time dependencies:
+
+| Package | Version range | License | Purpose |
+|---|---|---|---|
+| [tsdown](https://github.com/rolldown/tsdown) | `^0.22.14` | MIT | Build-time bundling of `lib/` (a regular dependency so the git-install channel's `prepare` script can build) |
+| [typescript](https://github.com/microsoft/TypeScript) | `^5.9.0` | Apache-2.0 | Build-time declaration emission (`lib/types/`) |
+| [zod](https://github.com/colinhacks/zod) | `^4.4.3` | MIT | Runtime value schema for the `data_quality` storage-domain table (bundled into `lib/`) |
+| [@deepseek-ai/cordis](https://www.npmjs.com/package/@deepseek-ai/cordis) | `^4.0.1` (peer) | See package | The plugin runtime |
+| [@deepseek-ai/schemastery](https://www.npmjs.com/package/@deepseek-ai/schemastery) | `^3.18.0` (peer) | See package | Configuration schema |
+| `@deepseek-ai/dsh-*` peers | `0.1.0-rc.6` (peer) | See packages | Official harness seams (`dsh-session`, `dsh-tools`, `dsh-storage`, `dsh-storage-domain`) |
+
+At runtime the plugin only talks to the harness services listed as
+peerDependencies; it performs no network requests and stores no credentials.

package/cordis.patch.yml

@@ -0,0 +1,46 @@
+# dsh-data-quality bundle patch: deterministic data profiling, cleaning, and
+# verification (ctx.dataQuality + data_profile / data_clean / data_verify).
+#
+# Every key below is a Config field (Schemastery schema); invalid values fail
+# the profile load loudly. See README.md "Configuration" for the full table.
+# The storage stack rows mirror dsh-fast: the plugin injects `storageDomain`,
+# so a bare profile needs the hub + a JSON backend + the domain form mounted.
+- insert:
+    - id: storage
+      name: '@deepseek-ai/dsh-storage'
+
+    - id: storage-json
+      name: '@deepseek-ai/dsh-storage-json'
+      config:
+        root: !!js dshHomePath('storages')
+
+    - id: storage-domain
+      name: '@deepseek-ai/dsh-storage-domain'
+      config:
+        backend: json
+
+    - id: dsh-data-quality
+      name: dsh-data-quality
+      config:
+        # Master switch. On by default; `false` mounts nothing at all.
+        enabled: true
+        # Hard row cap per dataset load; larger inputs are rejected loudly
+        # (use the tool's `sample` parameter to profile a subset).
+        maxRows: 200000
+        # Hard file-size cap in MiB per dataset load.
+        maxFileSizeMB: 64
+        # Default relative tolerance for numeric citation comparison when a
+        # citation omits `tolerance`.
+        defaultTolerance: 0.000000001
+        # Max evidence rows (verify failures) and preview rows (clean output)
+        # carried in one tool result.
+        evidenceRowLimit: 20
+        # Extensions the tools and the service accept as datasets.
+        allowedExtensions: ['.csv', '.tsv', '.json', '.jsonl']
+        # Absolute root for SERVICE-level calls (ctx.dataQuality.*) that carry
+        # no session workspace. Tool calls always use the session's workspace
+        # cwd instead. Empty = the harness process launch directory.
+        workspaceRoot: ''
+        # Persist profile/clean/verify reports to the `data_quality` storage
+        # domain (durable, queryable). Keys are returned as `reportKey`.
+        storeReports: true