vipii 0.1.0__tar.gz

vipii-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,27 @@
+name: CI
+
+on:
+  push:
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.9", "3.13"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install
+        run: |
+          python -m pip install --upgrade pip
+          python -m pip install -e ".[dev]"
+      - name: Ruff
+        run: |
+          ruff check .
+          ruff format --check .
+      - name: Pytest
+        run: pytest

vipii-0.1.0/.gitignore

@@ -0,0 +1,60 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+build/
+dist/
+downloads/
+eggs/
+.eggs/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# Test, coverage, and tooling caches
+.coverage
+.coverage.*
+.pytest_cache/
+.ruff_cache/
+.mypy_cache/
+.pyre/
+.tox/
+.nox/
+htmlcov/
+coverage.xml
+
+# Virtual environments
+.venv/
+venv/
+ENV/
+env/
+
+# Environment files
+.env
+.env.*
+!.env.example
+
+# IDE and editor files
+.vscode/
+.idea/
+*.swp
+*.swo
+*~
+
+# OS files
+.DS_Store
+Thumbs.db
+
+# Local example/runtime output
+*.log

vipii-0.1.0/AGENTS.md

@@ -0,0 +1,75 @@
+# Repository Guidelines
+
+## Project Shape
+
+- `vipii` is a Python package using a `src/` layout: package code lives in `src/vipii`.
+- The public API is exported from `src/vipii/__init__.py`.
+- Core modules:
+  - `models.py`: frozen dataclasses for `PIIMatch` and regex `Pattern`.
+  - `recognizers.py`: built-in Vietnamese structured PII recognizers, validators, and registry.
+  - `scoring.py`: context-window normalization and score boosting.
+  - `detector.py`: detector orchestration, overlap resolution, and redaction.
+  - `cli.py`: `argparse` CLI for `vipii scan`.
+  - `presidio.py`: optional Presidio adapter; importing the module should not require Presidio.
+
+## Coding Style
+
+- Target Python is `>=3.9`; keep compatibility with Python 3.9 through 3.13.
+- Use `from __future__ import annotations` in Python modules.
+- Prefer small, typed functions and dataclasses over large classes.
+- Use absolute imports from `vipii`, matching the existing modules.
+- Keep source formatted for Ruff with a 100-character line length.
+- Existing lint rules come from Ruff: `E`, `F`, `I`, `UP`, `B`, and `SIM`.
+- Keep user-facing text and file IO UTF-8 friendly; tests and examples contain Vietnamese text.
+- Use Any|Any for optional return in funciton instead of typing.Optional
+- At interface functions, place '...' instead left empty
+
+## Naming Patterns
+
+- PII entity labels are uppercase strings such as `CCCD`, `PHONE_NUMBER`, and `BANK_ACCOUNT`.
+- Recognizer names are lowercase snake_case such as `phone_number` and `vehicle_plate`.
+- Validators use `valid_*` names and return `bool`.
+- Helper functions use snake_case and are module-level unless they need object state.
+- CLI command functions are named around the action, for example `scan_input`, `scan_file`,
+  and `scan_text`.
+
+## Architecture Patterns
+
+- Built-in recognizers are regex `Pattern` objects plus optional validators and context words.
+- Scores start from `base_score` and are boosted by nearby context words in `scoring.py`.
+- `PIIDetector.detect()` gathers candidates from the registry, then resolves overlapping spans.
+- `PIIDetector.redact()` masks detected spans while preserving surrounding text.
+- Custom patterns are added through `PIIDetector.add_pattern()` and wrapped as a recognizer.
+- Optional dependencies should stay lazy, as in `presidio.py`.
+
+## Testing Style
+
+- Tests use pytest and live under `tests/`.
+- Prefer behavior-focused tests against the public API or CLI entry points.
+- CLI tests call `vipii.cli.main(...)` directly and assert stdout with `capsys`.
+- Use `tmp_path` for file-based CLI tests.
+- Fixture-driven detector coverage uses JSONL in `tests/fixtures/`.
+- Some tests use `# type: ignore[no-untyped-def]` for pytest fixtures without annotations.
+
+## Commands
+
+Install for development:
+
+```bash
+pip install -e ".[dev]"
+```
+
+Run lint and format checks:
+
+```bash
+ruff check .
+ruff format --check .
+```
+
+Run tests:
+
+```bash
+pytest
+```
+
+CI runs the same Ruff and pytest commands on Python 3.9 and 3.13.

vipii-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 vipii contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

vipii-0.1.0/PKG-INFO

@@ -0,0 +1,161 @@
+Metadata-Version: 2.4
+Name: vipii
+Version: 0.1.0
+Summary: Vietnamese PII detection with regex recognizers, validators, and context scoring.
+Author: vipii contributors
+License-Expression: MIT
+License-File: LICENSE
+Keywords: nlp,pii,privacy,redaction,vietnamese
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Text Processing :: Linguistic
+Requires-Python: >=3.9
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Requires-Dist: ruff>=0.5.0; extra == 'dev'
+Provides-Extra: ner
+Requires-Dist: torch>=2.0.0; extra == 'ner'
+Requires-Dist: transformers>=4.40.0; extra == 'ner'
+Provides-Extra: presidio
+Requires-Dist: presidio-analyzer>=2.2.0; extra == 'presidio'
+Description-Content-Type: text/markdown
+
+# vipii
+
+`vipii` is a Python library for detecting Vietnamese personally identifiable information (PII) in
+UTF-8 text. It combines deterministic regex-based recognizers, validator functions, overlap
+resolution, and Vietnamese context-window scoring to identify structured entities such as national
+IDs, phone numbers, tax codes, bank identifiers, passports, and vehicle plates.
+
+## Install
+
+```bash
+pip install vipii
+```
+
+For local development:
+
+```bash
+pip install -e ".[dev]"
+```
+
+## Python API
+
+```python
+from vipii import PIIDetector, Pattern
+
+detector = PIIDetector()
+detector.add_pattern(
+    Pattern(label="CUSTOMER_ID", regex=r"\bKH-\d{6}\b", context_words=["mã khách hàng"])
+)
+
+matches = detector.detect(
+    "Khách hàng Nguyễn Văn A, số điện thoại 0912 345 678, CCCD 001203000123."
+)
+
+for match in matches:
+    print(match.label, match.text, match.score)
+```
+
+## Optional NER
+
+Regex recognizers cover structured PII. For free-form names, locations, organizations, and addresses,
+enable an external Hugging Face token-classification model:
+
+```bash
+pip install "vipii[ner]"
+vipii scan "Nguyễn Văn A sống tại Hà Nội" --ner-model your-vietnamese-ner-model
+```
+
+```python
+from vipii import PIIDetector
+
+detector = PIIDetector(ner_model="your-vietnamese-ner-model")
+matches = detector.detect("Nguyễn Văn A sống tại Hà Nội")
+```
+
+The NER layer maps model labels such as `PER`, `LOC`, and `ORG` to `PERSON`, `LOCATION`, and
+`ORGANIZATION`. The model is not bundled; choose and evaluate one for your domain before production
+use.
+
+## CLI
+
+```bash
+vipii scan "Số điện thoại 0912 345 678 và CCCD 001203000123"
+vipii scan examples/customer_service.txt
+vipii scan examples/customer_service.txt --format json
+vipii scan examples/customer_service.txt --redact
+vipii scan "CCCD 001203000123" --redact
+vipii scan "Mã khách hàng KH-123456" --config examples/custom_recognizers.yml
+vipii scan "Nguyễn Văn A sống tại Hà Nội" --ner-model your-vietnamese-ner-model
+```
+
+## YAML recognizer config
+
+Built-in recognizers are loaded from `src/vipii/builtin_recognizers.yml`. You can append your own
+recognizers from a YAML file without writing Python:
+
+```yaml
+recognizers:
+  - name: customer_id
+    label: CUSTOMER_ID
+    patterns:
+      - regex: '\bKH-\d{6}\b'
+        context_words: ["mã khách hàng", "customer id"]
+        base_score: 0.6
+```
+
+Use `validator` only when you want one of vipii's built-in validators: `cccd`, `cmnd`, `phone`,
+`tax_code`, `bank_card`, `bank_account`, `passport`, or `vehicle_plate`.
+
+## Built-in recognizers
+
+- `CCCD` and `CMND`
+- `PHONE_NUMBER`
+- `MST`
+- `BANK_CARD`
+- `BANK_ACCOUNT`
+- `PASSPORT`
+- `VEHICLE_PLATE`
+
+The recognizers intentionally favor clear structured PII plus nearby Vietnamese context words such as
+`số điện thoại`, `cccd`, `mã số thuế`, and `biển số xe`. Names and free-form addresses can be handled
+by the optional NER layer.
+
+## Development
+
+```bash
+pip install -e ".[dev]"
+ruff check .
+ruff format --check .
+pytest
+```
+
+## Publishing
+
+Build and inspect the package before uploading:
+
+```bash
+python -m pip install --upgrade build twine
+python -m build
+python -m twine check dist/*
+```
+
+Upload to TestPyPI first:
+
+```bash
+python -m twine upload --repository testpypi dist/*
+```
+
+Then upload the same checked artifacts to PyPI:
+
+```bash
+python -m twine upload dist/*
+```

vipii-0.1.0/README.md

@@ -0,0 +1,132 @@
+# vipii
+
+`vipii` is a Python library for detecting Vietnamese personally identifiable information (PII) in
+UTF-8 text. It combines deterministic regex-based recognizers, validator functions, overlap
+resolution, and Vietnamese context-window scoring to identify structured entities such as national
+IDs, phone numbers, tax codes, bank identifiers, passports, and vehicle plates.
+
+## Install
+
+```bash
+pip install vipii
+```
+
+For local development:
+
+```bash
+pip install -e ".[dev]"
+```
+
+## Python API
+
+```python
+from vipii import PIIDetector, Pattern
+
+detector = PIIDetector()
+detector.add_pattern(
+    Pattern(label="CUSTOMER_ID", regex=r"\bKH-\d{6}\b", context_words=["mã khách hàng"])
+)
+
+matches = detector.detect(
+    "Khách hàng Nguyễn Văn A, số điện thoại 0912 345 678, CCCD 001203000123."
+)
+
+for match in matches:
+    print(match.label, match.text, match.score)
+```
+
+## Optional NER
+
+Regex recognizers cover structured PII. For free-form names, locations, organizations, and addresses,
+enable an external Hugging Face token-classification model:
+
+```bash
+pip install "vipii[ner]"
+vipii scan "Nguyễn Văn A sống tại Hà Nội" --ner-model your-vietnamese-ner-model
+```
+
+```python
+from vipii import PIIDetector
+
+detector = PIIDetector(ner_model="your-vietnamese-ner-model")
+matches = detector.detect("Nguyễn Văn A sống tại Hà Nội")
+```
+
+The NER layer maps model labels such as `PER`, `LOC`, and `ORG` to `PERSON`, `LOCATION`, and
+`ORGANIZATION`. The model is not bundled; choose and evaluate one for your domain before production
+use.
+
+## CLI
+
+```bash
+vipii scan "Số điện thoại 0912 345 678 và CCCD 001203000123"
+vipii scan examples/customer_service.txt
+vipii scan examples/customer_service.txt --format json
+vipii scan examples/customer_service.txt --redact
+vipii scan "CCCD 001203000123" --redact
+vipii scan "Mã khách hàng KH-123456" --config examples/custom_recognizers.yml
+vipii scan "Nguyễn Văn A sống tại Hà Nội" --ner-model your-vietnamese-ner-model
+```
+
+## YAML recognizer config
+
+Built-in recognizers are loaded from `src/vipii/builtin_recognizers.yml`. You can append your own
+recognizers from a YAML file without writing Python:
+
+```yaml
+recognizers:
+  - name: customer_id
+    label: CUSTOMER_ID
+    patterns:
+      - regex: '\bKH-\d{6}\b'
+        context_words: ["mã khách hàng", "customer id"]
+        base_score: 0.6
+```
+
+Use `validator` only when you want one of vipii's built-in validators: `cccd`, `cmnd`, `phone`,
+`tax_code`, `bank_card`, `bank_account`, `passport`, or `vehicle_plate`.
+
+## Built-in recognizers
+
+- `CCCD` and `CMND`
+- `PHONE_NUMBER`
+- `MST`
+- `BANK_CARD`
+- `BANK_ACCOUNT`
+- `PASSPORT`
+- `VEHICLE_PLATE`
+
+The recognizers intentionally favor clear structured PII plus nearby Vietnamese context words such as
+`số điện thoại`, `cccd`, `mã số thuế`, and `biển số xe`. Names and free-form addresses can be handled
+by the optional NER layer.
+
+## Development
+
+```bash
+pip install -e ".[dev]"
+ruff check .
+ruff format --check .
+pytest
+```
+
+## Publishing
+
+Build and inspect the package before uploading:
+
+```bash
+python -m pip install --upgrade build twine
+python -m build
+python -m twine check dist/*
+```
+
+Upload to TestPyPI first:
+
+```bash
+python -m twine upload --repository testpypi dist/*
+```
+
+Then upload the same checked artifacts to PyPI:
+
+```bash
+python -m twine upload dist/*
+```

vipii-0.1.0/examples/_helpers.py

@@ -0,0 +1,16 @@
+"""Shared helpers for example scripts."""
+
+from __future__ import annotations
+
+from vipii import PIIDetector
+
+
+def print_matches(title: str, text: str, detector: PIIDetector) -> None:
+    print(f"\n{title}")
+    print("-" * len(title))
+    print(text)
+    for match in detector.detect(text):
+        print(
+            f"{match.label:<14} {match.start:>2}:{match.end:<2} "
+            f"score={match.score:.2f} text={match.text!r}"
+        )

vipii-0.1.0/examples/basic_detection.py

@@ -0,0 +1,26 @@
+"""Detect built-in Vietnamese structured PII.
+
+Run from the repository after installing the package:
+
+    python examples/basic_detection.py
+"""
+
+from __future__ import annotations
+
+from _helpers import print_matches
+
+from vipii import PIIDetector
+
+
+def main() -> None:
+    detector = PIIDetector()
+    text = (
+        "Khách hàng có số điện thoại 0912 345 678, CCCD 001203000123, "
+        "mã số thuế 0312345678 và biển số xe 51F-123.45."
+    )
+
+    print_matches("Built-in structured PII", text, detector)
+
+
+if __name__ == "__main__":
+    main()

vipii-0.1.0/examples/concurrent_detection.py

@@ -0,0 +1,80 @@
+"""Run recognizers concurrently.
+
+Run from the repository after installing the package:
+
+    python examples/concurrent_detection.py
+"""
+
+from __future__ import annotations
+
+import re
+from dataclasses import dataclass
+from time import perf_counter, sleep
+
+from _helpers import print_matches
+
+from vipii import PIIDetector
+from vipii.models import PIIMatch
+
+
+@dataclass
+class SlowRegexRecognizer:
+    name: str
+    label: str
+    regex: str
+    delay_seconds: float = 0.5
+
+    def recognize(self, text: str) -> list[PIIMatch]:
+        sleep(self.delay_seconds)
+        return [
+            PIIMatch(
+                label=self.label,
+                start=match.start(),
+                end=match.end(),
+                text=match.group(0),
+                score=0.8,
+                recognizer=self.name,
+            )
+            for match in re.finditer(self.regex, text)
+        ]
+
+
+def timed_detect(title: str, detector: PIIDetector, text: str) -> None:
+    started = perf_counter()
+    matches = detector.detect(text)
+    elapsed = perf_counter() - started
+
+    print(f"\n{title}")
+    print("-" * len(title))
+    print(f"Detected {len(matches)} matches in {elapsed:.2f}s")
+    for match in matches:
+        print(
+            f"{match.label:<14} {match.start:>2}:{match.end:<2} "
+            f"score={match.score:.2f} text={match.text!r}"
+        )
+
+
+def main() -> None:
+    text = "Mã khách hàng KH-123456 có mã đơn hàng DH-98765."
+    recognizers = [
+        SlowRegexRecognizer("customer_id", "CUSTOMER_ID", r"\bKH-\d{6}\b"),
+        SlowRegexRecognizer("order_id", "ORDER_ID", r"\bDH-\d{5}\b"),
+    ]
+
+    concurrent_detector = PIIDetector(
+        recognizers=recognizers,
+        include_builtins=False,
+    )
+    sequential_detector = PIIDetector(
+        recognizers=recognizers,
+        include_builtins=False,
+        max_workers=1,
+    )
+
+    print_matches("Concurrent recognizers", text, concurrent_detector)
+    timed_detect("Concurrent timing", concurrent_detector, text)
+    timed_detect("Sequential timing", sequential_detector, text)
+
+
+if __name__ == "__main__":
+    main()

vipii-0.1.0/examples/custom_only.py

@@ -0,0 +1,24 @@
+"""Run a detector with only custom recognizers.
+
+Run from the repository after installing the package:
+
+    python examples/custom_only.py
+"""
+
+from __future__ import annotations
+
+from _helpers import print_matches
+
+from vipii import Pattern, PIIDetector
+
+
+def main() -> None:
+    detector = PIIDetector(include_builtins=False)
+    detector.add_pattern(Pattern(label="ORDER_ID", regex=r"\bDH-\d{5}\b"))
+    text = "Đơn hàng DH-12345 của số điện thoại 0912345678."
+
+    print_matches("Custom-only detector", text, detector)
+
+
+if __name__ == "__main__":
+    main()

vipii-0.1.0/examples/custom_pattern.py

@@ -0,0 +1,31 @@
+"""Add a custom regex pattern in Python.
+
+Run from the repository after installing the package:
+
+    python examples/custom_pattern.py
+"""
+
+from __future__ import annotations
+
+from _helpers import print_matches
+
+from vipii import Pattern, PIIDetector
+
+
+def main() -> None:
+    detector = PIIDetector()
+    detector.add_pattern(
+        Pattern(
+            label="CUSTOMER_ID",
+            regex=r"\bKH-\d{6}\b",
+            context_words=["mã khách hàng", "customer id"],
+            base_score=0.6,
+        )
+    )
+    text = "Mã khách hàng KH-123456 có số điện thoại 0912 345 678."
+
+    print_matches("Custom pattern from Python", text, detector)
+
+
+if __name__ == "__main__":
+    main()

vipii-0.1.0/examples/custom_recognizers.yml

@@ -0,0 +1,7 @@
+recognizers:
+  - name: customer_id
+    label: CUSTOMER_ID
+    patterns:
+      - regex: '\bKH-\d{6}\b'
+        context_words: ["mã khách hàng", "customer id"]
+        base_score: 0.6

vipii-0.1.0/examples/customer_service.txt

@@ -0,0 +1,6 @@
+Lưu ý: toàn bộ dữ liệu trong ví dụ này là giả.
+
+Khách hàng Trần Văn Minh gọi lên tổng đài, số điện thoại 090 123 4567.
+Bạn ấy cung cấp CCCD 001203000123 và mã số thuế cá nhân 0312345678.
+Nhân viên ghi nhận biển số xe 51F-123.45 và hộ chiếu B1234567.
+Thông tin thanh toán có thẻ 9704 0000 1234 5678 và tài khoản ngân hàng 123456789012 tại ngân hàng thử nghiệm.