@stll/anonymize 1.5.0 → 2.0.0-alpha.1

package/ATTRIBUTION.md

@@ -0,0 +1,70 @@
+# Attribution
+
+This library builds on ideas and patterns from several open-source
+projects and academic research.
+
+## Prior Art
+
+### Microsoft Presidio (Apache 2.0)
+
+- Context-word boosting architecture
+- Structured PII pattern design (IBAN, phone, email)
+- Operator concept (replace vs redact)
+- https://github.com/microsoft/presidio
+
+### GLiNER / GLiNER.js (MIT)
+
+- Span-level and token-level NER via ONNX
+- The `gliner/` module is an original implementation informed
+  by the GLiNER architecture (arXiv:2311.08526)
+- Processor and decoder logic reimplemented from scratch
+- https://github.com/urchade/GLiNER
+
+### NameTag / MorphoDiTa (ÚFAL, Charles University)
+
+- Czech NER and morphological analysis research
+- Czech name declension suffix patterns
+- https://ufal.mff.cuni.cz/nametag
+- https://ufal.mff.cuni.cz/morphodita
+
+### Text Anonymization Benchmark (NorskRegnesentral, MIT)
+
+- ECHR court case evaluation methodology
+- Entity type taxonomy for legal documents
+- https://github.com/NorskRegnesentral/text-anonymization-benchmark
+
+### Unicode CLDR
+
+- Multilingual month name data
+- https://cldr.unicode.org (Unicode License)
+
+## Deny List Data Sources
+
+### FinNLP/humannames (MIT)
+
+- ~195,000 person names (global, multilingual)
+- Used in: `dictionaries/names/global.json`
+- https://github.com/FinNLP/humannames
+- License: MIT
+
+### GeoNames (CC BY 4.0)
+
+- City and place names from the GeoNames gazetteer
+- Population threshold: ≥1,000 inhabitants
+- Includes native names, ASCII transliterations, and
+  alternate names across languages
+- Used in: `dictionaries/cities/*.json`
+- https://www.geonames.org
+- License: Creative Commons Attribution 4.0 International
+
+### Wikidata (CC0 1.0)
+
+- Courts, banks, insurance companies, government ministries,
+  universities, hospitals, and EU institutions
+- Labels and alternate labels in cs, sk, de, en
+- Used in: `dictionaries/courts/`, `dictionaries/banks/`,
+  `dictionaries/insurance/`, `dictionaries/government/`,
+  `dictionaries/education/`, `dictionaries/healthcare/`,
+  `dictionaries/international/`
+- https://www.wikidata.org
+- License: Creative Commons CC0 1.0 Universal

package/README.md

@@ -1,5 +1,5 @@
 <p align="center">
-  <img src="../../.github/assets/banner.png" alt="Stella anonymize" width="100%" />
+  <img src="../../.github/assets/banner.png" alt="stella anonymize" width="100%" />
 </p>
 
 # @stll/anonymize
@@ -16,50 +16,96 @@ bun add @stll/anonymize
 bun add @stll/anonymize-data
 ```
 
-For browser targets, install `@stll/anonymize-wasm` instead. It exposes the same runtime API through WebAssembly and is the supported entrypoint for Vite-based bundles.
+The Node.js package is Rust-native. Browser/WASM support is maintained through
+`@stll/anonymize-wasm` while the native core remains the v1 product path.
 
-## Usage
+## Usage: Node.js native SDK
 
 ```ts
-import { runPipeline } from "@stll/anonymize";
+import {
+  availableDefaultNativePipelineLanguages,
+  getDefaultNativePipeline,
+} from "@stll/anonymize/native-node";
+
+const languages = availableDefaultNativePipelineLanguages();
+const anonymizer = getDefaultNativePipeline(
+  languages.includes("en") ? { language: "en" } : {},
+);
+const result = anonymizer.redact_text(text);
+
+console.log(result.redaction.redactedText);
+```
 
-const entities = await runPipeline({
-  fullText: text,
-  config: {
-    labels: [
-      "person",
-      "organization",
-      "address",
-      "date",
-      "iban",
-      "phone number",
-    ],
-    threshold: 0.5,
-    enableRegex: true,
-    enableTriggerPhrases: true,
-    enableLegalForms: true,
-    enableNameCorpus: true,
-    enableDenyList: false,
-    enableGazetteer: false,
-    enableNer: false,
-    enableConfidenceBoost: true,
-    enableCoreference: true,
-    workspaceId: "default",
-  },
-  gazetteerEntries: [],
-});
+Call `getDefaultNativePipeline()` once during service startup and reuse the returned anonymizer. The package ships with a prepared native package, so the normal request path avoids rebuilding search automata. Use `preloadDefaultNativePipeline()` or `preloadDefaultNativePipelineAsync()` when the first document should not pay lazy regex warm-up.
+
+If your deployment knows the document language up front, select a scoped package at startup. The build emits `en`, `cs`, and `de` scoped packages by default, and `STELLA_ANONYMIZE_NATIVE_PACKAGE_LANGUAGES` can replace that list or be set to an empty value to build only the all-language package:
+
+```bash
+STELLA_ANONYMIZE_NATIVE_PACKAGE_LANGUAGES=en,cs,fr bun run build
 ```
 
-## Caller-owned deny lists and regexes
+```ts
+const anonymizer = getDefaultNativePipeline({ language: "en" });
+```
+
+Regional codes use the exact package when present and otherwise fall back to
+the base language package, so `en-US` can use the shipped `en` artifact.
+
+For build-time generated packages or caller-owned data, prepare the package before runtime and load the bytes in the process that handles documents.
 
-Use `customDenyList` for exact terms and variants that you control. These are matched by the deny-list layer, so keep `enableDenyList: true`.
+```bash
+bunx stella-anonymize-build-native-package \
+  --config ./anonymize-native-config.mjs \
+  --out ./dist/anonymize.stlanonpkg
+```
 
 ```ts
-const entities = await runPipeline({
-  fullText: text,
+import { load_prepared_package_file } from "@stll/anonymize/native-node";
+
+const anonymizer = load_prepared_package_file("./dist/anonymize.stlanonpkg");
+anonymizer.warmLazyRegex();
+const warmDiagnosticsJson = anonymizer.warmLazyRegexDiagnosticsJson();
+const result = anonymizer.redact_text(text, { redactString: "***" });
+```
+
+The config module may export a `PipelineConfig` directly or `{ config, gazetteerEntries }`. Include `@stll/anonymize-data` dictionaries there if your runtime config uses the deny-list or name-corpus layers; keep the corresponding layers enabled for caller-owned `customDenyList`, `customRegexes`, and gazetteers. Those inputs are part of the prepared package and should be regenerated when they change.
+
+## Python SDK
+
+```py
+import stella_anonymize as anonymize
+
+languages = anonymize.available_default_native_pipeline_languages()
+prepared = anonymize.preload_default_native_pipeline(
+    language="en" if "en" in languages else None
+)
+result = prepared.redact_text(text, redact_string="***")
+
+print(result.redaction.redacted_text)
+```
+
+The Python SDK uses the same Rust core and prepared-package contract as the Node SDK. Prefer `get_default_native_pipeline()`, `preload_default_native_pipeline()`, `load_prepared_package()`, or `load_prepared_package_file()` for repeated calls; top-level `redact_text()` and `redact_text_json()` prepare from config on each call.
+
+## Caller-Owned Deny Lists and Regexes
+
+Use `customDenyList` for exact terms and variants that you control. Use
+`customRegexes` for deterministic patterns that are not built into the package.
+Caller-owned data is part of the prepared package, so build or load a package
+from that config before serving documents.
+
+```ts
+import {
+  createNativePipelineFromConfig,
+  loadNativeAnonymizeBinding,
+} from "@stll/anonymize/native-node";
+
+const binding = loadNativeAnonymizeBinding();
+const pipeline = await createNativePipelineFromConfig({
+  binding,
   config: {
     ...baseConfig,
     enableDenyList: true,
+    enableRegex: true,
     customDenyList: [
       {
         value: "Project Nebula",
@@ -67,19 +113,6 @@ const entities = await runPipeline({
         label: "organization",
       },
     ],
-  },
-  gazetteerEntries: [],
-});
-```
-
-Use `customRegexes` for deterministic patterns that are not built into the package. These are matched by the regex layer, so keep `enableRegex: true`.
-
-```ts
-const entities = await runPipeline({
-  fullText: text,
-  config: {
-    ...baseConfig,
-    enableRegex: true,
     customRegexes: [
       {
         pattern: "\\bSTLL-[0-9]{4}\\b",
@@ -90,6 +123,8 @@ const entities = await runPipeline({
   },
   gazetteerEntries: [],
 });
+
+const result = pipeline.redactText(text);
 ```
 
 ## Browser setup
@@ -106,10 +141,13 @@ export default {
 
 ## Notes
 
+- Native architecture and extension guidance:
+  [`ARCHITECTURE.md`](ARCHITECTURE.md).
 - `labels: []` disables deterministic label filtering; when NER is enabled it falls back to the default label set.
 - `enableNameCorpus` also controls whether first names, surnames, and titles are injected into deny-list matching when `enableDenyList` is enabled.
-- The optional `@stll/anonymize-data` package carries the published dictionary and trigger data used by the deny-list layer.
-- `customDenyList` and `customRegexes` are part of the pipeline config and are included in the internal search cache key.
+- The optional `@stll/anonymize-data` package carries the published dictionary and trigger data used when building prepared packages.
+- `customDenyList` and `customRegexes` are part of the prepared package input and should be regenerated when they change.
+- The old TypeScript pipeline is kept only as temporary internal migration/test scaffolding under `src/legacy.ts`; it is not the product runtime.
 
 ## Built on
 

package/dist/address-boundaries.mjs

@@ -6,6 +6,7 @@ var address_boundaries_default = {
 		"jednajícím",
 		"jejímž jménem",
 		"kontaktní osoba",
+		"pokud",
 		"zapsán",
 		"zapsaná",
 		"zapsané",
@@ -36,6 +37,7 @@ var address_boundaries_default = {
 		"shall govern",
 		"shall be governed",
 		"to be enforced",
+		"with a copy",
 		"with the intention",
 		"without reference",
 		"without regard"

package/dist/address-jurisdiction-prefixes.mjs

@@ -0,0 +1,16 @@
+//#region src/data/address-jurisdiction-prefixes.json
+var _comment = "Address-like jurisdiction prefixes that are valid location/address spans without digits or street-type words. Lowercased and organized per language.";
+var en = [
+	"commonwealth of",
+	"district of",
+	"state of",
+	"territory of"
+];
+var address_jurisdiction_prefixes_default = {
+	_comment,
+	en
+};
+//#endregion
+export { _comment, address_jurisdiction_prefixes_default as default, en };
+
+//# sourceMappingURL=address-jurisdiction-prefixes.mjs.map

package/dist/address-jurisdiction-prefixes.mjs.map

@@ -0,0 +1 @@
+{"version":3,"file":"address-jurisdiction-prefixes.mjs","names":[],"sources":["../src/data/address-jurisdiction-prefixes.json"],"sourcesContent":[""],"mappings":""}

package/dist/address-stop-keywords.mjs

@@ -17,6 +17,16 @@ var cs = [
 	"ičo",
 	"ič"
 ];
+var de = [
+	"bank",
+	"bic",
+	"iban",
+	"steuer-id",
+	"steueridentifikationsnummer",
+	"steuernummer",
+	"ust-idnr",
+	"ust-idnr."
+];
 var en = [
 	"e-mail",
 	"email",
@@ -123,6 +133,7 @@ var it = [
 var address_stop_keywords_default = {
 	_comment,
 	cs,
+	de,
 	en,
 	pl,
 	fr,
@@ -132,6 +143,6 @@ var address_stop_keywords_default = {
 	it
 };
 //#endregion
-export { _comment, cs, address_stop_keywords_default as default, en, es, fr, it, pl, pt_br as "pt-br", sk };
+export { _comment, cs, de, address_stop_keywords_default as default, en, es, fr, it, pl, pt_br as "pt-br", sk };
 
 //# sourceMappingURL=address-stop-keywords.mjs.map

package/dist/address-unit-abbreviations.mjs

@@ -0,0 +1,15 @@
+//#region src/data/address-unit-abbreviations.json
+var address_unit_abbreviations_default = {
+	_comment: "Dotted address unit abbreviations that should not terminate address seed expansion. Organised per language because abbreviations are locale-specific.",
+	en: [
+		"apt.",
+		"bldg.",
+		"fl.",
+		"ste.",
+		"unit."
+	]
+};
+//#endregion
+export { address_unit_abbreviations_default as default };
+
+//# sourceMappingURL=address-unit-abbreviations.mjs.map

package/dist/address-unit-abbreviations.mjs.map

@@ -0,0 +1 @@
+{"version":3,"file":"address-unit-abbreviations.mjs","names":[],"sources":["../src/data/address-unit-abbreviations.json"],"sourcesContent":[""],"mappings":""}

package/dist/clause-noun-heads.mjs

@@ -33,7 +33,11 @@ var cs = [
 	"přílohu",
 	"dodatek",
 	"dodatku",
-	"oznámení"
+	"článek",
+	"oznámení",
+	"podmínky",
+	"předmět",
+	"ustanovení"
 ];
 var de = [
 	"vertrag",

package/dist/coreference-org-determiners.mjs

@@ -0,0 +1,19 @@
+//#region src/data/coreference-org-determiners.json
+var _comment = "Organization reference determiners used before propagated bare organization names. Values are regex fragments grouped by language.";
+var cs = ["společnost(?:i|í|em|u)?", "spolecnost(?:i|em|u)?"];
+var de = ["die\\s+(?:gesellschaft|firma)"];
+var en = ["the\\s+(?:company|corporation|firm)"];
+var es = ["la\\s+(?:empresa|sociedad)", "el\\s+(?:empresa|sociedad)"];
+var fr = ["la\\s+société"];
+var coreference_org_determiners_default = {
+	_comment,
+	cs,
+	de,
+	en,
+	es,
+	fr
+};
+//#endregion
+export { _comment, cs, de, coreference_org_determiners_default as default, en, es, fr };
+
+//# sourceMappingURL=coreference-org-determiners.mjs.map

package/dist/coreference-org-determiners.mjs.map

@@ -0,0 +1 @@
+{"version":3,"file":"coreference-org-determiners.mjs","names":[],"sources":["../src/data/coreference-org-determiners.json"],"sourcesContent":[""],"mappings":""}

package/dist/defined-term-heads.mjs

@@ -0,0 +1,15 @@
+//#region src/data/defined-term-heads.json
+var _comment = "Common head nouns for capitalized defined/legal concepts. These are not person names by themselves; detector-specific filters assemble this vocabulary where needed. Lowercased and organized per language.";
+var en = [
+	"association",
+	"period",
+	"reform"
+];
+var defined_term_heads_default = {
+	_comment,
+	en
+};
+//#endregion
+export { _comment, defined_term_heads_default as default, en };
+
+//# sourceMappingURL=defined-term-heads.mjs.map

package/dist/defined-term-heads.mjs.map

@@ -0,0 +1 @@
+{"version":3,"file":"defined-term-heads.mjs","names":[],"sources":["../src/data/defined-term-heads.json"],"sourcesContent":[""],"mappings":""}

package/dist/false-positive-shapes.mjs

@@ -0,0 +1,36 @@
+//#region src/data/false-positive-shapes.json
+var _comment = "Language-keyed lexical markers used by false-positive shape guards.";
+var addressComponentTerms = { "cs": [
+	"č.p.",
+	"č.ev.",
+	"č.",
+	"sídliště"
+] };
+var ambiguousStreetTypeTerms = { "fr": ["cours"] };
+var numberAbbrevPrefixes = {
+	"cs": ["čís.", "č."],
+	"de": ["nr."],
+	"en": ["no.", "n."]
+};
+var documentHeadingOrdinalMarkers = {
+	"cs": ["č.", "č"],
+	"de": ["nr.", "nr"],
+	"en": [
+		"no.",
+		"no",
+		"n.",
+		"n"
+	],
+	"global": ["#"]
+};
+var false_positive_shapes_default = {
+	_comment,
+	addressComponentTerms,
+	ambiguousStreetTypeTerms,
+	numberAbbrevPrefixes,
+	documentHeadingOrdinalMarkers
+};
+//#endregion
+export { _comment, addressComponentTerms, ambiguousStreetTypeTerms, false_positive_shapes_default as default, documentHeadingOrdinalMarkers, numberAbbrevPrefixes };
+
+//# sourceMappingURL=false-positive-shapes.mjs.map

package/dist/false-positive-shapes.mjs.map

@@ -0,0 +1 @@
+{"version":3,"file":"false-positive-shapes.mjs","names":[],"sources":["../src/data/false-positive-shapes.json"],"sourcesContent":[""],"mappings":""}