@dev-pi2pie/word-counter 0.1.5-canary.1 → 0.1.5-canary.3

package/README.md

@@ -37,6 +37,8 @@ For local development in this repository:
 ```bash
 git clone https://github.com/dev-pi2pie/word-counter.git
 cd word-counter
+rustup target add wasm32-unknown-unknown
+cargo install wasm-pack --locked
 bun install
 bun run build
 npm link
@@ -94,6 +96,24 @@ word-counter --han-language zh-Hant "漢字測試"
 word-counter --han-tag zh-Hans "汉字测试"
 ```
 
+Enable the optional WASM detector for ambiguous Latin and Han routes:
+
+```bash
+word-counter --detector wasm "This sentence should clearly be detected as English for the wasm detector path."
+word-counter --detector wasm "漢字測試需要更多內容才能觸發偵測"
+```
+
+Detector mode notes:
+
+- `--detector regex` is the default behavior.
+- `--detector wasm` only runs for ambiguous `und-Latn` and `und-Hani` chunks.
+- `--detector regex` keeps the original script/regex chunk-first detection path.
+- `--detector wasm` uses a detector-oriented ambiguous-window scoring pass before accepted tags are projected back onto the counting chunks.
+- In `--detector wasm` mode, Latin hint rules and explicit Latin hint flags are deferred until after detector evaluation and only relabel unresolved `und-Latn` output.
+- Very short chunks stay on the original `und-*` fallback.
+- Low-confidence or unsupported detector results fall back to `und-*`.
+- Technical-noise-heavy Latin windows stay conservative and may remain `und-Latn` even when the detector produces a wrong-but-confident language guess.
+
 Collect non-words (emoji/symbols/punctuation):
 
 ```bash
@@ -267,14 +287,24 @@ word-counter --path ./examples/test-case-multi-files-support --debug --verbose
 
 Use `--debug-report [path]` to route debug diagnostics to a JSONL report file:
 
-- no path: writes to current working directory with pattern `wc-debug-YYYYMMDD-HHmmss-<pid>.jsonl`
+- no path: writes to current working directory with pattern `wc-debug-YYYYMMDD-HHmmss-utc-<pid>.jsonl`
+- no path with `--detector-evidence`: writes with pattern `wc-detector-evidence-YYYYMMDD-HHmmss-utc-<pid>.jsonl`
 - path provided: writes to the specified location
 - default-name collision handling: appends `-<n>` suffix to avoid overwriting existing files
 - explicit path validation: existing directories are rejected (explicit paths are treated as file targets)
+- compatibility note: the autogenerated filename moved from the older local-time pattern to the new UTC `...-utc-...jsonl` pattern
 
 By default with `--debug-report`, debug lines are file-only (not mirrored to terminal).
 Use `--debug-report-tee` (alias: `--debug-tee`) to mirror to both file and `stderr`.
-Flag dependencies: `--verbose` requires `--debug`; `--debug-report` requires `--debug`; `--debug-report-tee`/`--debug-tee` requires `--debug-report`.
+Flag dependencies: `--verbose` requires `--debug`; `--detector-evidence` requires `--debug` and `--detector wasm`; `--debug-report` requires `--debug`; `--debug-report-tee`/`--debug-tee` requires `--debug-report`.
+
+Use `--detector-evidence` to add per-window detector evidence onto the same debug stream:
+
+- only meaningful with `--detector wasm`
+- compact mode emits bounded single-line previews plus detector decision metadata
+- verbose mode emits full raw detector windows and full normalized samples
+- evidence remains detector-window based even when output mode changes to `collector`, `char`, or another counting mode
+- fallback evidence reports the post-fallback final tag used by downstream counting output; in rare split-relabel cases it may also include `finalLocales`
 
 Examples:
 
@@ -283,16 +313,26 @@ word-counter --path ./examples/test-case-multi-files-support --debug --debug-rep
 word-counter --path ./examples/test-case-multi-files-support --debug --debug-report ./logs/debug.jsonl
 word-counter --path ./examples/test-case-multi-files-support --debug --debug-report ./logs/debug.jsonl --debug-report-tee
 word-counter --path ./examples/test-case-multi-files-support --debug --debug-report ./logs/debug.jsonl --debug-tee
+word-counter --detector wasm --debug --detector-evidence "This sentence should clearly be detected as English for the wasm detector path."
+word-counter --detector wasm --debug --verbose --detector-evidence "This sentence should clearly be detected as English for the wasm detector path."
+word-counter --detector wasm --debug --detector-evidence --debug-report
 ```
 
 Skip details stay debug-gated and can be suppressed with `--quiet-skips`.
 
+When `--format json` is combined with `--debug`, debug-only diagnostics are emitted under `debug.*`:
+
+- single input and merged batch may include `debug.detector`
+- per-file batch may include `debug.skipped`, `debug.detector`, and per-entry `files[i].debug.detector`
+- per-file top-level `skipped` is still emitted temporarily for compatibility
+
 ## How It Works
 
 - The runtime inspects each character's Unicode script to infer its likely locale tag (e.g., `und-Latn`, `und-Hani`, `ja`).
 - Adjacent characters that share the same locale tag are grouped into a chunk.
 - Each chunk is counted with `Intl.Segmenter` at `granularity: "word"`, caching segmenters to avoid re-instantiation.
 - Per-locale counts are summed into an overall total and printed to stdout.
+- With `--detector wasm`, ambiguous `und-Latn` and `und-Hani` chunks can be relabeled through the optional WASM detector before counting; unresolved `und-Latn` chunks then fall back to the existing Latin hint rules and explicit Latin hint precedence.
 
 ## Locale vs Language Code
 
@@ -316,6 +356,10 @@ import wordCounter, {
   segmentTextByLocale,
   showSingularOrPluralWord,
 } from "@dev-pi2pie/word-counter";
+import {
+  wordCounterWithDetector,
+  segmentTextByLocaleWithDetector,
+} from "@dev-pi2pie/word-counter/detector";
 
 wordCounter("Hello world", { latinLanguageHint: "en" });
 wordCounter("Hello world", { latinTagHint: "en" });
@@ -329,6 +373,11 @@ wordCounter("Hi 👋, world!", { mode: "char", nonWords: true });
 wordCounter("飛鳥 bird 貓 cat", { mode: "char-collector" });
 wordCounter("Hi\tthere\n", { nonWords: true, includeWhitespace: true });
 countCharsForLocale("👋", "en");
+await wordCounterWithDetector(
+  "This sentence should clearly be detected as English for the wasm detector path.",
+  { detector: "wasm" },
+);
+await segmentTextByLocaleWithDetector("Hello 世界", { detector: "regex" });
 ```
 
 Note: `includeWhitespace` only affects results when `nonWords: true` is enabled.
@@ -362,6 +411,7 @@ Sample output (with `nonWords: true` and `includeWhitespace: true`):
 
 ```js
 const wordCounter = require("@dev-pi2pie/word-counter");
+const detector = require("@dev-pi2pie/word-counter/detector");
 const {
   countCharsForLocale,
   countWordsForLocale,
@@ -383,6 +433,10 @@ wordCounter("Hi 👋, world!", { mode: "char", nonWords: true });
 wordCounter("飛鳥 bird 貓 cat", { mode: "char-collector" });
 wordCounter("Hi\tthere\n", { nonWords: true, includeWhitespace: true });
 countCharsForLocale("👋", "en");
+await detector.wordCounterWithDetector(
+  "This sentence should clearly be detected as English for the wasm detector path.",
+  { detector: "wasm" },
+);
 ```
 
 Note: `includeWhitespace` only affects results when `nonWords: true` is enabled.
@@ -437,6 +491,18 @@ Sample output (with `nonWords: true` and `includeWhitespace: true`):
 | -------------------------- | -------- | ------------------------------ |
 | `showSingularOrPluralWord` | function | Formats singular/plural words. |
 
+#### Detector Subpath
+
+Import from `@dev-pi2pie/word-counter/detector` for the explicit detector-enabled API.
+
+| Export                        | Kind     | Notes                                           |
+| ----------------------------- | -------- | ----------------------------------------------- |
+| `wordCounterWithDetector`     | function | Async detector-aware counting entrypoint.       |
+| `segmentTextByLocaleWithDetector` | function | Async detector-aware locale segmentation.  |
+| `countSectionsWithDetector`   | function | Async detector-aware section counting.          |
+| `DEFAULT_DETECTOR_MODE`       | value    | Current default detector mode (`regex`).        |
+| `DETECTOR_MODES`              | value    | Supported detector modes.                       |
+
 #### Types
 
 | Export                 | Kind | Notes                                             |
@@ -650,6 +716,10 @@ Example JSON (trimmed):
 
 - Detection is regex/script based, not statistical language-ID.
 - Ambiguous Latin defaults to `und-Latn`; Han fallback defaults to `und-Hani`.
+- `--detector wasm` is optional and conservative; it only runs for ambiguous chunks that meet minimum script-bearing length thresholds.
+- In `--detector wasm` mode, ambiguous Latin stays on `und-Latn` for detector eligibility first, then built-in/custom Latin rules and explicit Latin hints are applied only if the detector leaves that chunk unresolved.
+- The current first WASM engine is `whatlang`, remapped into this package's public tags.
+- The npm package ships one portable WASM artifact; users do not install per-OS detector packages.
 - Use explicit tag and hint flags when you need deterministic tagging.
 - Full notes (built-in heuristics, limitations, and override guidance) are tracked in `docs/locale-tag-detection-notes.md`.