dsh-cait 1.0.4 → 1.1.3

package/README.md

@@ -16,7 +16,7 @@
 <p align="center">
   <a href="https://www.npmjs.com/package/dsh-cait"><img alt="npm version" src="https://img.shields.io/npm/v/dsh-cait.svg" /></a>
   <img alt="DSH 0.2.x" src="https://img.shields.io/badge/DSH-0.2.x-4d6bfe.svg" />
-  <img alt="Tests: 13 suites" src="https://img.shields.io/badge/tests-13%20suites-brightgreen.svg" />
+  <img alt="Tests: 29 suites" src="https://img.shields.io/badge/tests-29%20suites-brightgreen.svg" />
   <img alt="No API key required" src="https://img.shields.io/badge/API%20key-not%20required-6b7280.svg" />
 </p>
 
@@ -34,10 +34,10 @@ translated document.
 | | |
 |---|---|
 | **Native to DSH** | A real sidebar panel, a main workbench, DSH theme tokens, and the same store for humans and agents. No iframe, no separate app. |
-| **TM that explains itself** | Multiple TM libraries, mounted-only lookup, exact / near-exact / high / medium / low fuzzy grades, and automatic write-back on confirm. |
+| **TM that explains itself** | Multiple TM libraries, mounted-only lookup, 101% context (ICE) matches, exact / near-exact / high / medium / low fuzzy grades, and automatic write-back on confirm. |
 | **Glossary-aware editing** | Multiple term bases, source-language filtering, inline highlighting, click-to-insert terms, and per-term case sensitivity. |
 | **A document workflow** | Tabs, close/reopen, recent-document cleanup, duplicate-import reuse, and progress per document. |
-| **Real format coverage** | TXT, Markdown, DOCX, TMX, CSV, SRT/VTT, PO, XLIFF 1.2/2.0/SDLXLIFF, HTML, JSON, XLSX, PPTX, and RTF. |
+| **Real format coverage** | TXT, Markdown, DOCX, EPUB, TMX, CSV, SRT/VTT, PO, XLIFF 1.2/2.0/SDLXLIFF, HTML, JSON, XLSX, PPTX, and RTF. |
 | **Agent-ready** | Seven tools share the open document and store, so an agent can translate alongside you. |
 
 ## Quick start
@@ -60,6 +60,10 @@ manual reload.
 ### Translation memory
 
 - Multiple TM libraries with id, name, mount state, and a default write library.
+- **Context match (ICE) at 101%.** Each confirmation records the two neighbouring
+  source sentences. A later segment with the same neighbours scores 101% and is
+  graded `context`; identical text with different neighbours stays at 100%. A key
+  keeps up to 8 context variants and always keeps the context-free one.
 - Exact, near-exact, and three fuzzy bands:
   - `near_exact` compares a key that ignores digits, punctuation, and whitespace;
     it does not depend on a score cutoff.
@@ -74,7 +78,8 @@ manual reload.
   `bootstrap().tm.count` is the mounted count and `bootstrap().tm.total` is the
   all-library count. Use `cat_lookup_tm` or the library modal for full data.
 - TMX 1.4 `<tu>/<tuv>/<seg>` import/export. Inline TMX placeholders are flattened
-  to plain text with an explicit warning.
+  to plain text with an explicit warning. Context travels as
+  `<prop type="x-context">`, so a CAIT round-trip keeps 101% matches.
 
 ### Glossary
 
@@ -83,11 +88,71 @@ manual reload.
 - CSV/TSV import, per-term notes, source-language filtering, and a
   `caseSensitive` flag honored by both the workbench and `cat_lookup_glossary`.
 - Language codes are canonicalized (`EN` → `en`, `ZH_cn` → `zh-CN`).
+- Entries can be edited or deleted in any library, mounted or not; mount state
+  only decides what the workbench searches and writes.
+- **Terminology mining**: *Mine terms from the document* in the glossary toolbar
+  lists candidates with occurrence counts, sample wording from memory, and a
+  marker when the term is already known. Counting finds the candidates and ranks
+  them (Latin n-grams, repeated CJK substrings filtered by boundary entropy);
+  a model only judges a shortlist, which is the part models are good at.
+
+### Library workspace
+
+`Documents` / `Libraries` / `Align` in the topbar switches between the translator
+and the library manager. The library view needs no open document.
+
+- One table for translation memory, one for glossaries, with paging
+  (20/50/100/200, remembered), language filters, and a search that can be scoped
+  to any combination of libraries.
+- A side panel edits the selected entry in place, adds entries, and shows when a
+  draft is unsaved.
+- Translation memory exports to **TMX**; glossaries export to **TBX**, **CSV**, or
+  **JSON**. Import and clear stay available in the same toolbar.
+- The right panel's library button opens this workspace with the panel's search
+  text already applied, so a lookup in the panel continues in the full list.
+
+### Alignment
+
+`Documents` / `Libraries` / `Align` — the third view builds a translation
+memory from two finished files: the source document and its translation, as
+separate files in any supported format and any pair of languages.
+
+- Cross-language scoring: shared anchors (numbers, `{0}` placeholders, URLs,
+  capitalised names), length normalised per language, and paragraph structure.
+  Word overlap is deliberately not used, because it cannot compare two languages.
+- Blocked dynamic programming pairs 1:1, 1:0, 0:1, 2:1 and 1:2; rows are graded
+  high / to review / doubtful and start selected only when they are high.
+- Each row can be merged with the next, split on line breaks, or removed before
+  writing; the target library is chosen in the toolbar, and identical entries are
+  skipped rather than duplicated.
 
 ### Workbench
 
 - Paragraph/sentence segmentation with a per-segment target editor.
-- Draft saving, confirm-and-next, previous/next navigation, and progress.
+- Segment states: **pending** → **translated** → **confirmed**, with the origin of
+  each translation (`tm`, `mt`, `agent`, `manual`) kept alongside the text. The
+  rail filters by state and search covers source, target, and suggestions.
+- **Segment paging**: the rail pages through the filtered segments with a page
+  size of 20/50/100/200 and a box to jump straight to a segment number. Jumping
+  clears the filter when the target segment would otherwise be hidden.
+- **Pre-translate** fills the whole document from exact TM matches and machine
+  translation in small blocks, shows `done/total`, and can be stopped between
+  blocks. Confirmed and already-translated segments are never overwritten.
+- **Stacked review**: source, the machine/agent baseline, and the editable target
+  are laid out top to bottom, so each block gets the full pane width.
+  **Confirm all** accepts every segment awaiting review after one confirmation
+  dialog.
+- Draft saving, confirm-and-next, previous/next navigation, and progress. The
+  selected segment and the draft are persisted (debounced) so a restart resumes
+  where you left off.
+- **Repeated segments are mirrored**: confirming a sentence fills every other
+  unconfirmed copy with the same source (`origin: auto`), editing a draft keeps
+  them in sync, and *Fill repeated segments* does the whole document at once.
+  Confirmed segments are never touched.
+- Advisory QA markers in the rail and an export-time summary: missing
+  placeholders/tags, mismatched numbers, unused glossary terms, empty targets,
+  an untouched copy of the source, and one source with several translations.
+  Nothing is blocked and nothing is stored.
 - Document tabs; closing a tab does not delete the document.
 - **Open existing document** reopens a closed document.
 - Interface language switch in **Settings → Interface**: *Follow DSH language*
@@ -100,9 +165,25 @@ manual reload.
 - Re-importing identical content in the same format reuses the existing document
   instead of creating a duplicate. The fingerprint covers languages, source,
   format, and exchange metadata.
-- Responsive density: the workbench measures its DSH panel and applies a
-  `--cait-u` multiplier of 1×–1.8×. A 1600×900 panel renders at ~1.32× and a
-  2560×1440 panel at 1.8×; short windows stay at 1× rather than shrinking.
+- Responsive density: the workbench measures its DSH panel and applies a damped
+  `--cait-u` multiplier of 1×–1.45×. A 1600×900 panel renders at ~1.17× and a
+  maximized 4K panel stops at 1.45×, which keeps large windows close to the host
+  UI and leaves room for the three review columns.
+- The workbench follows the host light/dark theme, including native controls
+  (`color-scheme`), so selects, checkboxes, and scrollbars match the panel.
+
+### Preprocessing
+
+Imported source text passes through a preprocessing stage before segmentation:
+
+- Collapse repeated spaces and tabs, trim indentation, and merge consecutive
+  blank lines.
+- Remove zero-width characters and stray control characters.
+- Placeholder tokens (`{1}`, `{2}`, …), punctuation, casing, and paragraph order
+  are left untouched, so XLIFF/HTML/EPUB inline markup still round-trips.
+- The three steps are independently switchable in **Settings → Preprocessing** and
+  are persisted with the other settings. They apply to new imports and
+  `createDocument` calls; existing documents are not rewritten.
 
 ### MT assistance
 
@@ -125,15 +206,16 @@ provider/model/code/status; the client receives only `MT_PROVIDER_FAILED`.
 | Format | Import | Export | Notes |
 | --- | :---: | :---: | --- |
 | TXT / Markdown | ✅ | ✅ | Paragraph and sentence segmentation |
-| CSV | ✅ | ✅ | Optional source/target column by header name or 1-based number; existing bilingual export |
+| CSV | ✅ | ✅ | Comma, semicolon, tab, or pipe delimiter auto-detected; source/target column by header name or 1-based number; existing bilingual export |
 | DOCX | ✅ | ✅ | Body paragraphs; table blocks are excluded from import. Export patches the original package, keeping styles, numbering, images, headers, tables, and layout. |
-| TMX 1.4 | ✅ | ✅ | Common `<tu>/<tuv>/<seg>` subset; inline markup flattened with warning |
+| EPUB | ✅ | ✅ | EPUB 3 spine order; XHTML text blocks with inline markup kept as tokens and patched back into the retained package. DRM-protected, fixed-layout, and scripted EPUBs are not supported. |
+| TMX 1.4 | ✅ | ✅ | Common `<tu>/<tuv>/<seg>` subset; inline markup flattened with warning; context kept in `<prop type="x-context">` |
 | SRT / WebVTT | ✅ | ✅ | One cue per segment, timings/settings/identifier retained; semantic tags kept; invalid ranges rejected |
 | PO | ✅ | ✅ | Contexts, plural forms, translator comments, obsolete `#~` entries |
 | XLIFF 1.2 | ✅ | ✅ | Inline tags become `{n}` tokens; missing tokens block confirmation |
 | XLIFF 2.0 | ✅ | ✅ | Basic `<unit><segment><source>/<target>` subset |
 | SDLXLIFF | ✅ | ✅ | Basic XLIFF 1.2 path; SDL-specific extensions are not round-tripped |
-| HTML | ✅ | ✅ | Text blocks only; original markup/CSS not preserved |
+| HTML | ✅ | ✅ | Text blocks and inline tags preserved; document CSS/layout is not rebuilt |
 | JSON | ✅ | ✅ | String-path subset; object/array structure preserved via a template; keys and numbers are not translated |
 | XLSX | ✅ | ✅ | First worksheet; `Source`/`Target` headers or columns A/B; export is a new two-column workbook |
 | PPTX | ✅ | ✅* | Source-preserving export; requires the imported `.pptx` package. Body text only; notes/charts/SmartArt not imported. |
@@ -143,8 +225,28 @@ provider/model/code/status; the client receives only `MT_PROVIDER_FAILED`.
 is missing rather than silently producing an unformatted deck.
 
 Exchange metadata (subtitle timings, PO contexts/plurals/comments, XLIFF ids and
-inline token maps, JSON paths/templates, XLSX rows, PPTX slide/paragraph positions)
-is stored on the document/segments and persists in the snapshot.
+inline token maps, JSON paths/templates, XLSX rows, PPTX slide/paragraph positions,
+EPUB spine hrefs and block indexes) is stored on the document/segments and persists
+in the snapshot.
+
+### PDF is not supported
+
+CAIT does not import PDF, and future releases will not add PDF support. PDF is a
+fixed-layout print format: it has no logical paragraphs, reading order, or inline
+markup. Text extracted from it is fragmented across drawing operations, loses
+headings, tables, and inline emphasis, and produces segments that are poor input
+for translation memory and MT — in other words, PDF is extremely unsuitable for
+AI-related work.
+
+To translate PDF content, convert it to a supported format first, for example:
+
+```sh
+pandoc input.pdf -t docx -o output.docx   # or -t html, -t markdown
+```
+
+Adobe Acrobat's **Export to Word/HTML**, LibreOffice, or any other converter that
+produces DOCX, HTML, EPUB, or plain text works too. Scanned PDFs need OCR before
+conversion; CAIT has no image OCR stage.
 
 ## Install
 
@@ -161,7 +263,7 @@ In DSH Desktop (or any profile with the `plugin_manager` tool):
 plugin_manager install_bundle target: dsh-cait
 ```
 
-Pin a release with `dsh-cait@1.0.4`, or ask for the newest one with
+Pin a release with `dsh-cait@1.1.3`, or ask for the newest one with
 `dsh-cait@latest`. The manager resolves the registry package, applies the bundle
 patch, and activates the plugin — no hand-written profile rows.
 
@@ -229,7 +331,7 @@ not both.
 ```
 
 The snapshot stores documents, open tabs, TM, glossary, libraries, mounts, and
-non-secret settings. Imported DOCX/PPTX source packages live in `sources/`.
+non-secret settings. Imported DOCX/PPTX/EPUB source packages live in `sources/`.
 
 In-app settings (**Settings**) live in the snapshot rather than in the YAML
 above: source/target language, fuzzy threshold, TM result limit, MT
@@ -242,8 +344,9 @@ confirmation. Other structural edits remain best-effort and surface
 
 If the snapshot is corrupt or has an unknown version, CAIT logs a warning, moves
 it to `store.json.corrupt-<timestamp>`, and starts empty instead of failing
-activation. Version-1 snapshots are migrated to version 2 with default TM and
-glossary libraries.
+activation. Version-1 and version-2 snapshots are migrated to version 3 with
+default TM and glossary libraries; a version-2 draft that had text but was never
+confirmed becomes `translated` with `origin: manual`.
 
 ## Agent tools
 
@@ -256,6 +359,20 @@ glossary libraries.
 | `cat_list_documents` | List documents and progress |
 | `cat_switch_document` | Select the active document |
 | `cat_suggest_mt` | MT via the selected DSH route, or the explicit offline fallback |
+| `cat_pre_translate` | Fill one block from TM + MT, skipping confirmed/translated segments; returns `next` |
+| `cat_get_segments` | Bounded segment window with status, origin, and suggestions |
+| `cat_confirm_segments` | Confirm a list of indices, a window, or everything awaiting review |
+| `cat_propose_translation` | Leave a suggestion plus a reason without touching the confirmed target |
+| `cat_align` | Align two texts and report high / to-review / doubtful pairs |
+| `cat_consistency_report` | Group repeated sources that carry different translations |
+| `cat_term_candidates` | Mining shortlist: frequency, sample sentences, existing memory wording |
+| `cat_add_terms` | Add accepted terminology to a library in one batch |
+
+An agent can therefore draft a whole document without clicking in the UI:
+`cat_pre_translate` with `from: next` until `next` is `null`, then
+`cat_get_segments` to review, then `cat_confirm_segments` to accept. For
+terminology it reads `cat_term_candidates`, judges the shortlist, and writes the
+result with `cat_add_terms`.
 
 The workbench and tools share one `CatStore`, so an agent can translate the same
 document the user has open.
@@ -266,7 +383,7 @@ document the user has open.
 flowchart LR
   UI["DSH Web UI<br/>(sidebar + workbench)"] --> Client["client.js<br/>generated Web bundle"]
   Client --> API["/cait/api/rpc<br/>same-origin JSON"]
-  Tools["7 agent tools"] --> Store["CatStore"]
+  Tools["15 agent tools"] --> Store["CatStore"]
   API --> Store
   Store --> Snapshot[("store.json<br/>+ sources/")]
 ```
@@ -301,11 +418,18 @@ cait/
 │  ├─ errors.js
 │  ├─ store-utils.js
 │  ├─ libraries.js          # TM/glossary registries + mounts
+│  ├─ library-view.js       # Library paging/search + export entry points
+│  ├─ library-export.js     # Glossary TBX / CSV / JSON builders
 │  ├─ tm.js
 │  ├─ glossary.js
 │  ├─ persistence.js
 │  ├─ import-export.js
 │  ├─ formats.js            # CSV/SRT/VTT/PO/XLIFF/HTML/JSON/RTF
+│  ├─ html.js               # Shared HTML/XHTML run scanner + patcher
+│  ├─ epub.js               # EPUB package read / spine import / patch export
+│  ├─ preprocess.js         # Import-time source cleaning stage
+│  ├─ pre-translate.js      # Block pre-translation methods
+│  ├─ qa.js                 # Derived QA report (never persisted)
 │  ├─ xlsx.js
 │  ├─ pptx.js
 │  ├─ segmenter.js
@@ -334,6 +458,8 @@ node --check client.js
 node _dev/test-core.mjs
 node _dev/test-imports.mjs
 node _dev/test-formats.mjs
+node _dev/test-epub.mjs
+node _dev/test-preprocess.mjs
 node _dev/test-second-tier.mjs
 node _dev/test-docx-export.mjs
 node _dev/test-api.mjs
@@ -344,11 +470,20 @@ node _dev/test-libraries.mjs
 node _dev/test-mounts.mjs
 node _dev/test-ui-scale.mjs
 node _dev/test-mt.mjs
+node _dev/test-mt-routes.mjs
+node _dev/test-three-state.mjs
+node _dev/test-pretranslate.mjs
+node _dev/test-qa.mjs
+node _dev/test-library-crud.mjs
+node _dev/test-library-view.mjs
+node _dev/test-pagination.mjs
+node _dev/test-theme-tokens.mjs
+node _dev/test-e2e-1.1.mjs
 ```
 
 The import/export fixtures are `_dev/sample.docx`, `_dev/sample-table.docx`, and
 `_dev/sample-format.docx`; regenerate them with the matching
-`python _dev/make-*.py` scripts. All 13 suites should pass.
+`python _dev/make-*.py` scripts. All 29 suites should pass.
 
 ### Releasing
 
@@ -392,7 +527,10 @@ GitHub with a release.
 
 ## Known limits and roadmap
 
-- No PDF, EPUB, or image OCR import yet.
+- No PDF import; PDF is intentionally out of scope (see
+  [PDF is not supported](#pdf-is-not-supported)). No image OCR yet.
+- EPUB import follows the spine and body XHTML text blocks; DRM-protected,
+  fixed-layout, scripted, and MathML-heavy books are not supported.
 - DOCX import skips tables, headers, footnotes, fields, alt text, and tracked
   deletions. DOCX export leaves those regions in the source language.
 - Mixed-run DOCX/PPTX paragraphs inherit the first run's formatting.
@@ -402,23 +540,34 @@ GitHub with a release.
 - HTML/RTF export does not round-trip original styling; JSON translates string
   values only.
 - TM matching is lexical, not semantic; no stemming, synonyms, or embeddings.
-- Library rename/delete and per-tab mount state are not part of v1.0.0.
+- One translation per source text per context: a repeated sentence keeps a
+  separate variant for each surrounding pair, capped at 8 variants per key, and
+  reuses the generic entry when the neighbours differ.
+- Context is the two neighbouring source sentences only; document, customer, or
+  project metadata is not part of the match.
+- TBX and JSON glossary exports are one-way; only TMX and CSV import back.
 - The workbench UI ships in Chinese, English, and Japanese, switchable in
   Settings; this README ships in English, Simplified Chinese, and Japanese, and
   the English version is the source of truth.
 
-Planned next: PDF and EPUB adapters, image/vision input through a multimodal DSH
-route, richer library management, style-preserving XLSX output, and XLIFF
-`<alt-trans>` / notes support.
+Planned next: image/vision input through a multimodal DSH route,
+style-preserving XLSX output, and XLIFF `<alt-trans>` / notes support.
 
 ## Changelog and versioning
 
 See [CHANGELOG.md](CHANGELOG.md). `v1.0.0` was the first public release;
 `v1.0.1` adds the Japanese UI and the in-app interface language switch,
 `v1.0.2` makes *Follow DSH language* honour the DSH locale, `v1.0.3`
-publishes the package to npm, and `v1.0.4` moves later releases to the
-provenance-publishing workflow. The project was developed as `0.1.0` and
-renumbered for the GitHub release.
+publishes the package to npm, `v1.0.4` moves later releases to the
+provenance-publishing workflow, `v1.0.5` adds EPUB, import preprocessing,
+and the PDF policy, `v1.1.0` adds agent-driven pre-translation, the
+three-state segment model, stacked review, and advisory QA, `v1.1.1` adds
+the library workspace with TMX/TBX/CSV/JSON export, segment paging, and the
+panel-to-library jump, `v1.1.2` adds context-aware memory with 101% (ICE)
+matches, and `v1.1.3` adds two-file alignment, automatic propagation of
+repeated segments, terminology mining, and the proposal primitive agents use.
+The project
+was developed as `0.1.0` and renumbered for the GitHub release.
 
 > **License:** Apache License 2.0 — see [LICENSE](LICENSE). Redistributions must
 > retain the attribution in [NOTICE](NOTICE).