@stll/anonymize 1.5.0 → 2.0.0

package/ATTRIBUTION.md

@@ -0,0 +1,70 @@
+# Attribution
+
+This library builds on ideas and patterns from several open-source
+projects and academic research.
+
+## Prior Art
+
+### Microsoft Presidio (Apache 2.0)
+
+- Context-word boosting architecture
+- Structured PII pattern design (IBAN, phone, email)
+- Operator concept (replace vs redact)
+- https://github.com/microsoft/presidio
+
+### GLiNER / GLiNER.js (MIT)
+
+- Span-level and token-level NER via ONNX
+- The `gliner/` module is an original implementation informed
+  by the GLiNER architecture (arXiv:2311.08526)
+- Processor and decoder logic reimplemented from scratch
+- https://github.com/urchade/GLiNER
+
+### NameTag / MorphoDiTa (ÚFAL, Charles University)
+
+- Czech NER and morphological analysis research
+- Czech name declension suffix patterns
+- https://ufal.mff.cuni.cz/nametag
+- https://ufal.mff.cuni.cz/morphodita
+
+### Text Anonymization Benchmark (NorskRegnesentral, MIT)
+
+- ECHR court case evaluation methodology
+- Entity type taxonomy for legal documents
+- https://github.com/NorskRegnesentral/text-anonymization-benchmark
+
+### Unicode CLDR
+
+- Multilingual month name data
+- https://cldr.unicode.org (Unicode License)
+
+## Deny List Data Sources
+
+### FinNLP/humannames (MIT)
+
+- ~195,000 person names (global, multilingual)
+- Used in: `dictionaries/names/global.json`
+- https://github.com/FinNLP/humannames
+- License: MIT
+
+### GeoNames (CC BY 4.0)
+
+- City and place names from the GeoNames gazetteer
+- Population threshold: ≥1,000 inhabitants
+- Includes native names, ASCII transliterations, and
+  alternate names across languages
+- Used in: `dictionaries/cities/*.json`
+- https://www.geonames.org
+- License: Creative Commons Attribution 4.0 International
+
+### Wikidata (CC0 1.0)
+
+- Courts, banks, insurance companies, government ministries,
+  universities, hospitals, and EU institutions
+- Labels and alternate labels in cs, sk, de, en
+- Used in: `dictionaries/courts/`, `dictionaries/banks/`,
+  `dictionaries/insurance/`, `dictionaries/government/`,
+  `dictionaries/education/`, `dictionaries/healthcare/`,
+  `dictionaries/international/`
+- https://www.wikidata.org
+- License: Creative Commons CC0 1.0 Universal

package/README.md

@@ -1,5 +1,5 @@
 <p align="center">
-  <img src="../../.github/assets/banner.png" alt="Stella anonymize" width="100%" />
+  <img src="../../.github/assets/banner.png" alt="stella anonymize" width="100%" />
 </p>
 
 # @stll/anonymize
@@ -16,50 +16,96 @@ bun add @stll/anonymize
 bun add @stll/anonymize-data
 ```
 
-For browser targets, install `@stll/anonymize-wasm` instead. It exposes the same runtime API through WebAssembly and is the supported entrypoint for Vite-based bundles.
+The Node.js package is Rust-native. Browser/WASM support is maintained through
+`@stll/anonymize-wasm`, which wraps the same native core.
 
-## Usage
+## Usage: Node.js native SDK
 
 ```ts
-import { runPipeline } from "@stll/anonymize";
+import {
+  availableDefaultNativePipelineLanguages,
+  getDefaultNativePipeline,
+} from "@stll/anonymize/native-node";
+
+const languages = availableDefaultNativePipelineLanguages();
+const anonymizer = getDefaultNativePipeline(
+  languages.includes("en") ? { language: "en" } : {},
+);
+const result = anonymizer.redact_text(text);
+
+console.log(result.redaction.redactedText);
+```
 
-const entities = await runPipeline({
-  fullText: text,
-  config: {
-    labels: [
-      "person",
-      "organization",
-      "address",
-      "date",
-      "iban",
-      "phone number",
-    ],
-    threshold: 0.5,
-    enableRegex: true,
-    enableTriggerPhrases: true,
-    enableLegalForms: true,
-    enableNameCorpus: true,
-    enableDenyList: false,
-    enableGazetteer: false,
-    enableNer: false,
-    enableConfidenceBoost: true,
-    enableCoreference: true,
-    workspaceId: "default",
-  },
-  gazetteerEntries: [],
-});
+Call `getDefaultNativePipeline()` once during service startup and reuse the returned anonymizer. The package ships with a prepared native package, so the normal request path avoids rebuilding search automata. Use `preloadDefaultNativePipeline()` or `preloadDefaultNativePipelineAsync()` when the first document should not pay lazy regex warm-up.
+
+If your deployment knows the document language up front, select a scoped package at startup. The build emits `en`, `cs`, and `de` scoped packages by default, and `STELLA_ANONYMIZE_NATIVE_PACKAGE_LANGUAGES` can replace that list or be set to an empty value to build only the all-language package:
+
+```bash
+STELLA_ANONYMIZE_NATIVE_PACKAGE_LANGUAGES=en,cs,fr bun run build
 ```
 
-## Caller-owned deny lists and regexes
+```ts
+const anonymizer = getDefaultNativePipeline({ language: "en" });
+```
+
+Regional codes use the exact package when present and otherwise fall back to
+the base language package, so `en-US` can use the shipped `en` artifact.
+
+For build-time generated packages or caller-owned data, prepare the package before runtime and load the bytes in the process that handles documents.
 
-Use `customDenyList` for exact terms and variants that you control. These are matched by the deny-list layer, so keep `enableDenyList: true`.
+```bash
+bunx stella-anonymize-build-native-package \
+  --config ./anonymize-native-config.mjs \
+  --out ./dist/anonymize.stlanonpkg
+```
 
 ```ts
-const entities = await runPipeline({
-  fullText: text,
+import { load_prepared_package_file } from "@stll/anonymize/native-node";
+
+const anonymizer = load_prepared_package_file("./dist/anonymize.stlanonpkg");
+anonymizer.warmLazyRegex();
+const warmDiagnosticsJson = anonymizer.warmLazyRegexDiagnosticsJson();
+const result = anonymizer.redact_text(text, { redactString: "***" });
+```
+
+The config module may export a `PipelineConfig` directly or `{ config, gazetteerEntries }`. Include `@stll/anonymize-data` dictionaries there if your runtime config uses the deny-list or name-corpus layers; keep the corresponding layers enabled for caller-owned `customDenyList`, `customRegexes`, and gazetteers. Those inputs are part of the prepared package and should be regenerated when they change.
+
+## Python SDK
+
+```py
+import stella_anonymize as anonymize
+
+languages = anonymize.available_default_native_pipeline_languages()
+prepared = anonymize.preload_default_native_pipeline(
+    language="en" if "en" in languages else None
+)
+result = prepared.redact_text(text, redact_string="***")
+
+print(result.redaction.redacted_text)
+```
+
+The Python SDK uses the same Rust core and prepared-package contract as the Node SDK. Prefer `get_default_native_pipeline()`, `preload_default_native_pipeline()`, `load_prepared_package()`, or `load_prepared_package_file()` for repeated calls; top-level `redact_text()` and `redact_text_json()` prepare from config on each call.
+
+## Caller-Owned Deny Lists and Regexes
+
+Use `customDenyList` for exact terms and variants that you control. Use
+`customRegexes` for deterministic patterns that are not built into the package.
+Caller-owned data is part of the prepared package, so build or load a package
+from that config before serving documents.
+
+```ts
+import {
+  createNativePipelineFromConfig,
+  loadNativeAnonymizeBinding,
+} from "@stll/anonymize/native-node";
+
+const binding = loadNativeAnonymizeBinding();
+const pipeline = await createNativePipelineFromConfig({
+  binding,
   config: {
     ...baseConfig,
     enableDenyList: true,
+    enableRegex: true,
     customDenyList: [
       {
         value: "Project Nebula",
@@ -67,19 +113,6 @@ const entities = await runPipeline({
         label: "organization",
       },
     ],
-  },
-  gazetteerEntries: [],
-});
-```
-
-Use `customRegexes` for deterministic patterns that are not built into the package. These are matched by the regex layer, so keep `enableRegex: true`.
-
-```ts
-const entities = await runPipeline({
-  fullText: text,
-  config: {
-    ...baseConfig,
-    enableRegex: true,
     customRegexes: [
       {
         pattern: "\\bSTLL-[0-9]{4}\\b",
@@ -90,6 +123,8 @@ const entities = await runPipeline({
   },
   gazetteerEntries: [],
 });
+
+const result = pipeline.redactText(text);
 ```
 
 ## Browser setup
@@ -106,10 +141,13 @@ export default {
 
 ## Notes
 
+- Native architecture and extension guidance:
+  [`ARCHITECTURE.md`](ARCHITECTURE.md).
 - `labels: []` disables deterministic label filtering; when NER is enabled it falls back to the default label set.
 - `enableNameCorpus` also controls whether first names, surnames, and titles are injected into deny-list matching when `enableDenyList` is enabled.
-- The optional `@stll/anonymize-data` package carries the published dictionary and trigger data used by the deny-list layer.
-- `customDenyList` and `customRegexes` are part of the pipeline config and are included in the internal search cache key.
+- The optional `@stll/anonymize-data` package carries the published dictionary and trigger data used when building prepared packages.
+- `customDenyList` and `customRegexes` are part of the prepared package input and should be regenerated when they change.
+- The old TypeScript pipeline is kept only as temporary internal migration/test scaffolding under `src/legacy.ts`; it is not the product runtime.
 
 ## Built on