allyanonimiser 0.1.0__tar.gz

allyanonimiser-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Stephen Oates
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

allyanonimiser-0.1.0/MANIFEST.in

@@ -0,0 +1,25 @@
+include LICENSE
+include README.md
+include requirements.txt
+
+recursive-include tests *.py
+recursive-include allyanonimiser *.py
+
+prune .git
+prune .github
+prune venv
+prune env
+prune dist
+prune build
+prune output
+prune temp_files
+prune anonymized_files
+prune challenging_dataset
+prune datasets
+prune cache
+
+global-exclude *.py[cod]
+global-exclude __pycache__
+global-exclude *.so
+global-exclude .DS_Store
+global-exclude CLAUDE.md

allyanonimiser-0.1.0/PKG-INFO

@@ -0,0 +1,252 @@
+Metadata-Version: 2.1
+Name: allyanonimiser
+Version: 0.1.0
+Summary: Australian-focused PII detection and anonymization for the insurance industry
+Home-page: https://github.com/srepho/Allyanonimiser
+Author: Stephen Oates
+Author-email: Stephen Oates <stephen.j.a.oates@gmail.com>
+License: MIT License
+        
+        Copyright (c) 2025 Stephen Oates
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+        
+Project-URL: Homepage, https://github.com/srepho/Allyanonimiser
+Project-URL: Bug Tracker, https://github.com/srepho/Allyanonimiser/issues
+Project-URL: Documentation, https://github.com/srepho/Allyanonimiser#readme
+Keywords: pii,anonymization,privacy,insurance,australia
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Security
+Classifier: Topic :: Text Processing
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+Provides-Extra: dev
+Provides-Extra: llm
+License-File: LICENSE
+
+# Allyanonimiser
+
+Australian-focused PII detection and anonymization for the insurance industry.
+
+## Features
+
+- **Australian-Specific PII Detection**: Specialized recognizers for Australian TFNs, Medicare numbers, driver's licenses, and other Australian-specific identifiers.
+- **Insurance Industry Focus**: Recognition of policy numbers, claim references, vehicle identifiers, and other insurance-specific data.
+- **Long Text Processing**: Optimized for processing lengthy free-text fields like claim notes, medical reports, and emails.
+- **Custom Pattern Creation**: Easy creation of custom entity recognizers for organization-specific data.
+- **Synthetic Data Generation**: Generate realistic Australian test data for validation.
+- **LLM Integration**: Use Language Models to create challenging datasets for testing.
+- **Extensible Architecture**: Built on Presidio and spaCy with a modular, extensible design.
+
+## Installation
+
+```bash
+# Install from PyPI
+pip install allyanonimiser
+
+# Install the required spaCy model
+python -m spacy download en_core_web_lg
+```
+
+## Quick Start
+
+```python
+from allyanonimiser import create_au_insurance_analyzer
+
+# Create an analyzer with Australian and insurance patterns
+analyzer = create_au_insurance_analyzer()
+
+# Analyze text
+results = analyzer.analyze(
+    text="Please reference your policy AU-12345678 for claims related to your vehicle rego XYZ123.",
+    language="en"
+)
+
+# Print results
+for result in results:
+    print(f"Entity: {result.entity_type}, Text: {result.text}, Score: {result.score}")
+```
+
+## Processing Insurance Documents
+
+### Claim Notes
+
+```python
+from allyanonimiser import analyze_claim_notes
+
+# Long claim note text
+claim_note = """
+Claim Details:
+Spoke with the insured John Smith (TFN: 123 456 789) regarding damage to his vehicle ABC123.
+The incident occurred on 14/05/2023 when another vehicle collided with the rear of his car.
+Policy number: POL-987654321
+
+Vehicle Details:
+Toyota Corolla 2020
+VIN: 1HGCM82633A123456
+Registration: ABC123
+
+Contact Information:
+Phone: 0412 345 678
+Email: john.smith@example.com
+Address: 123 Main St, Sydney NSW 2000
+"""
+
+# Analyze the claim note
+analysis = analyze_claim_notes(claim_note)
+
+# Access structured information
+print("Incident Description:", analysis["incident_description"])
+print("\nPII-rich segments:")
+for segment in analysis["pii_segments"]:
+    print(f"  - {segment['text'][:50]}... (PII likelihood: {segment['pii_likelihood']:.2f})")
+
+# Anonymize the text
+from allyanonimiser import EnhancedAnonymizer
+anonymizer = EnhancedAnonymizer(analyzer=create_au_insurance_analyzer())
+anonymized = anonymizer.anonymize(claim_note)
+print("\nAnonymized text:")
+print(anonymized["text"])
+```
+
+### Processing Emails
+
+```python
+from allyanonimiser.insurance import InsuranceEmailAnalyzer
+
+email_text = """
+From: adjuster@insurance.com.au
+To: customer@example.com
+Subject: Your Claim CL-12345678
+
+Dear Mr. Smith,
+
+Thank you for your recent claim submission regarding your vehicle (Registration: XYZ123).
+
+We have assigned your claim number CL-12345678. Please reference this number in all future correspondence.
+
+Your policy POL-9876543 covers this type of damage, and we'll need the following information:
+1. Your Medicare number
+2. Additional photos of the damage
+3. The repair quote from the mechanic
+
+Please call me at 03 9876 5432 if you have any questions.
+
+Kind regards,
+Sarah Johnson
+Claims Assessor
+"""
+
+email_analyzer = InsuranceEmailAnalyzer()
+analysis = email_analyzer.analyze(email_text)
+
+print("Email Subject:", analysis["subject"])
+print("Claim Number:", analysis["claim_number"])
+print("Policy Number:", analysis["policy_number"])
+print("Customer Name:", analysis["customer_name"])
+print("Identified PII:", analysis["pii_entities"])
+```
+
+## Creating Custom Patterns
+
+```python
+from allyanonimiser import CustomPatternDefinition, create_pattern_from_examples
+
+# Create a custom pattern for internal reference numbers
+internal_ref_examples = [
+    "Internal reference: REF-12345",
+    "Ref Number: REF-98765",
+    "Reference: REF-55555"
+]
+
+pattern = create_pattern_from_examples(
+    entity_type="INTERNAL_REFERENCE",
+    examples=internal_ref_examples,
+    context=["internal", "reference", "ref"],
+    pattern_type="regex"
+)
+
+# Add to an existing analyzer
+analyzer.add_pattern(pattern)
+```
+
+## Using the Pattern Registry
+
+```python
+from allyanonimiser import PatternRegistry, CustomPatternDefinition
+
+# Create a registry
+registry = PatternRegistry()
+
+# Register patterns
+registry.register_pattern(CustomPatternDefinition(
+    entity_type="BROKER_CODE",
+    patterns=["BRK-[0-9]{4}"],
+    context=["broker", "agent", "representative"],
+    name="broker_code_recognizer"
+))
+
+# Share patterns across applications
+registry.export_patterns("insurance_patterns.json")
+
+# Later, in another application
+registry.import_patterns("insurance_patterns.json")
+```
+
+## Working with Australian Data
+
+```python
+from allyanonimiser.patterns import get_au_pattern_definitions
+
+# Get all Australian pattern definitions
+au_patterns = get_au_pattern_definitions()
+
+# Print information about each pattern
+for pattern in au_patterns:
+    print(f"Entity Type: {pattern['entity_type']}")
+    print(f"Description: {pattern['description']}")
+    print(f"Example Patterns: {pattern['patterns'][:2]}")
+    print("Context Terms:", ", ".join(pattern['context'][:5]))
+    print()
+```
+
+## Generating Australian Test Data
+
+```python
+from allyanonimiser.generators import AustralianSyntheticDataGenerator
+
+# Create a data generator
+generator = AustralianSyntheticDataGenerator()
+
+# Generate a dataset of Australian insurance documents
+generator.generate_dataset(
+    num_documents=50,
+    output_dir="au_insurance_dataset",
+    include_annotations=True
+)
+```
+
+## License
+
+MIT License

allyanonimiser-0.1.0/README.md

@@ -0,0 +1,205 @@
+# Allyanonimiser
+
+Australian-focused PII detection and anonymization for the insurance industry.
+
+## Features
+
+- **Australian-Specific PII Detection**: Specialized recognizers for Australian TFNs, Medicare numbers, driver's licenses, and other Australian-specific identifiers.
+- **Insurance Industry Focus**: Recognition of policy numbers, claim references, vehicle identifiers, and other insurance-specific data.
+- **Long Text Processing**: Optimized for processing lengthy free-text fields like claim notes, medical reports, and emails.
+- **Custom Pattern Creation**: Easy creation of custom entity recognizers for organization-specific data.
+- **Synthetic Data Generation**: Generate realistic Australian test data for validation.
+- **LLM Integration**: Use Language Models to create challenging datasets for testing.
+- **Extensible Architecture**: Built on Presidio and spaCy with a modular, extensible design.
+
+## Installation
+
+```bash
+# Install from PyPI
+pip install allyanonimiser
+
+# Install the required spaCy model
+python -m spacy download en_core_web_lg
+```
+
+## Quick Start
+
+```python
+from allyanonimiser import create_au_insurance_analyzer
+
+# Create an analyzer with Australian and insurance patterns
+analyzer = create_au_insurance_analyzer()
+
+# Analyze text
+results = analyzer.analyze(
+    text="Please reference your policy AU-12345678 for claims related to your vehicle rego XYZ123.",
+    language="en"
+)
+
+# Print results
+for result in results:
+    print(f"Entity: {result.entity_type}, Text: {result.text}, Score: {result.score}")
+```
+
+## Processing Insurance Documents
+
+### Claim Notes
+
+```python
+from allyanonimiser import analyze_claim_notes
+
+# Long claim note text
+claim_note = """
+Claim Details:
+Spoke with the insured John Smith (TFN: 123 456 789) regarding damage to his vehicle ABC123.
+The incident occurred on 14/05/2023 when another vehicle collided with the rear of his car.
+Policy number: POL-987654321
+
+Vehicle Details:
+Toyota Corolla 2020
+VIN: 1HGCM82633A123456
+Registration: ABC123
+
+Contact Information:
+Phone: 0412 345 678
+Email: john.smith@example.com
+Address: 123 Main St, Sydney NSW 2000
+"""
+
+# Analyze the claim note
+analysis = analyze_claim_notes(claim_note)
+
+# Access structured information
+print("Incident Description:", analysis["incident_description"])
+print("\nPII-rich segments:")
+for segment in analysis["pii_segments"]:
+    print(f"  - {segment['text'][:50]}... (PII likelihood: {segment['pii_likelihood']:.2f})")
+
+# Anonymize the text
+from allyanonimiser import EnhancedAnonymizer
+anonymizer = EnhancedAnonymizer(analyzer=create_au_insurance_analyzer())
+anonymized = anonymizer.anonymize(claim_note)
+print("\nAnonymized text:")
+print(anonymized["text"])
+```
+
+### Processing Emails
+
+```python
+from allyanonimiser.insurance import InsuranceEmailAnalyzer
+
+email_text = """
+From: adjuster@insurance.com.au
+To: customer@example.com
+Subject: Your Claim CL-12345678
+
+Dear Mr. Smith,
+
+Thank you for your recent claim submission regarding your vehicle (Registration: XYZ123).
+
+We have assigned your claim number CL-12345678. Please reference this number in all future correspondence.
+
+Your policy POL-9876543 covers this type of damage, and we'll need the following information:
+1. Your Medicare number
+2. Additional photos of the damage
+3. The repair quote from the mechanic
+
+Please call me at 03 9876 5432 if you have any questions.
+
+Kind regards,
+Sarah Johnson
+Claims Assessor
+"""
+
+email_analyzer = InsuranceEmailAnalyzer()
+analysis = email_analyzer.analyze(email_text)
+
+print("Email Subject:", analysis["subject"])
+print("Claim Number:", analysis["claim_number"])
+print("Policy Number:", analysis["policy_number"])
+print("Customer Name:", analysis["customer_name"])
+print("Identified PII:", analysis["pii_entities"])
+```
+
+## Creating Custom Patterns
+
+```python
+from allyanonimiser import CustomPatternDefinition, create_pattern_from_examples
+
+# Create a custom pattern for internal reference numbers
+internal_ref_examples = [
+    "Internal reference: REF-12345",
+    "Ref Number: REF-98765",
+    "Reference: REF-55555"
+]
+
+pattern = create_pattern_from_examples(
+    entity_type="INTERNAL_REFERENCE",
+    examples=internal_ref_examples,
+    context=["internal", "reference", "ref"],
+    pattern_type="regex"
+)
+
+# Add to an existing analyzer
+analyzer.add_pattern(pattern)
+```
+
+## Using the Pattern Registry
+
+```python
+from allyanonimiser import PatternRegistry, CustomPatternDefinition
+
+# Create a registry
+registry = PatternRegistry()
+
+# Register patterns
+registry.register_pattern(CustomPatternDefinition(
+    entity_type="BROKER_CODE",
+    patterns=["BRK-[0-9]{4}"],
+    context=["broker", "agent", "representative"],
+    name="broker_code_recognizer"
+))
+
+# Share patterns across applications
+registry.export_patterns("insurance_patterns.json")
+
+# Later, in another application
+registry.import_patterns("insurance_patterns.json")
+```
+
+## Working with Australian Data
+
+```python
+from allyanonimiser.patterns import get_au_pattern_definitions
+
+# Get all Australian pattern definitions
+au_patterns = get_au_pattern_definitions()
+
+# Print information about each pattern
+for pattern in au_patterns:
+    print(f"Entity Type: {pattern['entity_type']}")
+    print(f"Description: {pattern['description']}")
+    print(f"Example Patterns: {pattern['patterns'][:2]}")
+    print("Context Terms:", ", ".join(pattern['context'][:5]))
+    print()
+```
+
+## Generating Australian Test Data
+
+```python
+from allyanonimiser.generators import AustralianSyntheticDataGenerator
+
+# Create a data generator
+generator = AustralianSyntheticDataGenerator()
+
+# Generate a dataset of Australian insurance documents
+generator.generate_dataset(
+    num_documents=50,
+    output_dir="au_insurance_dataset",
+    include_annotations=True
+)
+```
+
+## License
+
+MIT License