pdf-anonymizer-cli 0.3.1__tar.gz → 0.3.3__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- pdf_anonymizer_cli-0.3.3/PKG-INFO +144 -0
- pdf_anonymizer_cli-0.3.3/README.md +131 -0
- {pdf_anonymizer_cli-0.3.1 → pdf_anonymizer_cli-0.3.3}/pyproject.toml +1 -1
- {pdf_anonymizer_cli-0.3.1 → pdf_anonymizer_cli-0.3.3}/src/pdf_anonymizer_cli/cli.py +63 -41
- pdf_anonymizer_cli-0.3.3/src/pdf_anonymizer_cli.egg-info/PKG-INFO +144 -0
- pdf_anonymizer_cli-0.3.1/PKG-INFO +0 -103
- pdf_anonymizer_cli-0.3.1/README.md +0 -90
- pdf_anonymizer_cli-0.3.1/src/pdf_anonymizer_cli.egg-info/PKG-INFO +0 -103
- {pdf_anonymizer_cli-0.3.1 → pdf_anonymizer_cli-0.3.3}/setup.cfg +0 -0
- {pdf_anonymizer_cli-0.3.1 → pdf_anonymizer_cli-0.3.3}/src/pdf_anonymizer_cli/__init__.py +0 -0
- {pdf_anonymizer_cli-0.3.1 → pdf_anonymizer_cli-0.3.3}/src/pdf_anonymizer_cli/main.py +0 -0
- {pdf_anonymizer_cli-0.3.1 → pdf_anonymizer_cli-0.3.3}/src/pdf_anonymizer_cli.egg-info/SOURCES.txt +0 -0
- {pdf_anonymizer_cli-0.3.1 → pdf_anonymizer_cli-0.3.3}/src/pdf_anonymizer_cli.egg-info/dependency_links.txt +0 -0
- {pdf_anonymizer_cli-0.3.1 → pdf_anonymizer_cli-0.3.3}/src/pdf_anonymizer_cli.egg-info/entry_points.txt +0 -0
- {pdf_anonymizer_cli-0.3.1 → pdf_anonymizer_cli-0.3.3}/src/pdf_anonymizer_cli.egg-info/requires.txt +0 -0
- {pdf_anonymizer_cli-0.3.1 → pdf_anonymizer_cli-0.3.3}/src/pdf_anonymizer_cli.egg-info/top_level.txt +0 -0
|
@@ -0,0 +1,144 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: pdf-anonymizer-cli
|
|
3
|
+
Version: 0.3.3
|
|
4
|
+
Summary: CLI for a tool to anonymize PDF, Markdown, and plain text files using LLMs.
|
|
5
|
+
Author-email: Leonid Ganeline <leo.gan.57@gmail.com>
|
|
6
|
+
License: MIT
|
|
7
|
+
Project-URL: repository, https://github.com/leo-gan/anonymizer
|
|
8
|
+
Requires-Python: >=3.10
|
|
9
|
+
Description-Content-Type: text/markdown
|
|
10
|
+
Requires-Dist: typer
|
|
11
|
+
Requires-Dist: rich
|
|
12
|
+
Requires-Dist: pdf-anonymizer-core
|
|
13
|
+
|
|
14
|
+
# 🦉🫥 PDF Anonymizer CLI
|
|
15
|
+
|
|
16
|
+
A command-line interface for anonymizing PDF, Markdown, and plain text files using LLMs.
|
|
17
|
+
|
|
18
|
+
- **High-Quality Anonymization**: Leverages LLMs to identify and replace Personally Identifiable Information (PII) with high accuracy.
|
|
19
|
+
- **Large File Support**: Consistently anonymizes large files (tested up to 1GB).
|
|
20
|
+
- **Multi-Provider & Cost-Effective**: Free to use with local [Ollama](https://ollama.com/) models. It also supports major providers like [OpenAI](https://openai.com/), [Anthropic](https://www.anthropic.com/), [Google](https://ai.google.com/), [Hugging Face](https://huggingface.co/), and [OpenRouter](https://openrouter.ai/).
|
|
21
|
+
- **Reversible**: Supports deanonymization to recover original data when needed.
|
|
22
|
+
- **Multi-Format**: Works with PDF, Markdown, and plain text files.
|
|
23
|
+
|
|
24
|
+
|
|
25
|
+
## Installation
|
|
26
|
+
|
|
27
|
+
Install the CLI with your favorite package manager. To use a specific LLM provider, you must install the corresponding extra.
|
|
28
|
+
|
|
29
|
+
- **Google**: `pip install "pdf-anonymizer-cli[google]"`
|
|
30
|
+
- **Ollama**: `pip install "pdf-anonymizer-cli[ollama]"`
|
|
31
|
+
- **Hugging Face**: `pip install "pdf-anonymizer-cli[huggingface]"`
|
|
32
|
+
- **OpenRouter**: `pip install "pdf-anonymizer-cli[openrouter]"`
|
|
33
|
+
- **OpenAI**: `pip install "pdf-anonymizer-cli[openai]"`
|
|
34
|
+
- **Anthropic**: `pip install "pdf-anonymizer-cli[anthropic]"`
|
|
35
|
+
|
|
36
|
+
You can also install multiple extras at once:
|
|
37
|
+
|
|
38
|
+
```bash
|
|
39
|
+
pip install "pdf-anonymizer-cli[google,openrouter]"
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
This installs the `pdf-anonymizer` executable.
|
|
43
|
+
|
|
44
|
+
|
|
45
|
+
## Environment Variables
|
|
46
|
+
|
|
47
|
+
The CLI will automatically load a `.env` file from the current directory or any parent directory. For consistency, it's recommended to place a single `.env` file at the root of the repository.
|
|
48
|
+
|
|
49
|
+
- `GOOGLE_API_KEY`: Required when using Google models.
|
|
50
|
+
- `HUGGING_FACE_TOKEN`: Required when using Hugging Face models. You can get a token from [here](https://huggingface.co/docs/hub/security-tokens).
|
|
51
|
+
- `OPENROUTER_API_KEY`: Required when using OpenRouter models.
|
|
52
|
+
- `OPENAI_API_KEY`: Required when using OpenAI models.
|
|
53
|
+
- `ANTHROPIC_API_KEY`: Required when using Anthropic models.
|
|
54
|
+
- `OLLAMA_HOST`: Optional, defaults to `http://localhost:11434` when using Ollama models.
|
|
55
|
+
|
|
56
|
+
Example `.env` file:
|
|
57
|
+
```env
|
|
58
|
+
GOOGLE_API_KEY="YOUR_API_KEY_HERE"
|
|
59
|
+
HUGGING_FACE_TOKEN="YOUR_HF_TOKEN_HERE"
|
|
60
|
+
OPENROUTER_API_KEY="YOUR_OPENROUTER_KEY"
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
## Usage
|
|
64
|
+
|
|
65
|
+
### Anonymize
|
|
66
|
+
|
|
67
|
+
The `run` command anonymizes one or more files.
|
|
68
|
+
|
|
69
|
+
```bash
|
|
70
|
+
pdf-anonymizer run FILE_PATH [FILE_PATH ...] \
|
|
71
|
+
[--characters-to-anonymize INTEGER] \
|
|
72
|
+
[--prompt-name {simple|detailed}] \
|
|
73
|
+
[--model-name TEXT] \
|
|
74
|
+
[--anonymized-entities PATH]
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
**Arguments**:
|
|
78
|
+
- `FILE_PATH`: Path to one or several PDF, Markdown, or text files for anonymization.
|
|
79
|
+
|
|
80
|
+
**Options**:
|
|
81
|
+
- `--characters-to-anonymize INTEGER`: Number of characters to process in each chunk (default: `100000`).
|
|
82
|
+
- `--prompt-name [simple|detailed]`: The prompt template to use (default: `detailed`).
|
|
83
|
+
- `--model-name TEXT`: The language model to use.
|
|
84
|
+
- `--anonymized-entities PATH`: Path to a file with a list of entities to anonymize.
|
|
85
|
+
|
|
86
|
+
**Models**:
|
|
87
|
+
You can use any of the predefined models below, or specify a new model using the format `"provider/model-name"`.
|
|
88
|
+
For example: `--model-name "google/gemini-flash-latest"`.
|
|
89
|
+
|
|
90
|
+
- **Google**: `gemini-2.5-pro`, `gemini-2.5-flash` (default), `gemini-2.5-flash-lite`.
|
|
91
|
+
- **Ollama**: `gemma:7b`, `phi4-mini`.
|
|
92
|
+
- **Hugging Face**: `openai/gpt-oss-20b`, `mistralai/Mistral-7B-Instruct-v0.1`, `HuggingFaceH4/zephyr-7b-beta`.
|
|
93
|
+
- **OpenRouter**: `openai/gpt-4o`, `google/gemini-pro`.
|
|
94
|
+
- **OpenAI**: `gpt-4o`, `gpt-5`.
|
|
95
|
+
- **Anthropic**: `claude-4-sonet`, `claude-4.5-sonet`.
|
|
96
|
+
|
|
97
|
+
### Examples
|
|
98
|
+
|
|
99
|
+
**Basic anonymization with the default model (Google)**:
|
|
100
|
+
```bash
|
|
101
|
+
pdf-anonymizer run document.pdf
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
**A new model (Google) and a simple prompt**:
|
|
105
|
+
```bash
|
|
106
|
+
pdf-anonymizer run notes.md --model-name "google/gemini-flash-latest" --prompt-name simple
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
**Using an OpenRouter model**:
|
|
110
|
+
```bash
|
|
111
|
+
pdf-anonymizer run report.pdf --model-name "openai/gpt-4o"
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
### Deanonymize
|
|
115
|
+
|
|
116
|
+
The `deanonymize` command reverts anonymization using a mapping file.
|
|
117
|
+
|
|
118
|
+
```bash
|
|
119
|
+
pdf-anonymizer deanonymize ANONYMIZED_FILE MAPPING_FILE
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
**Arguments**:
|
|
123
|
+
- `ANONYMIZED_FILE`: Path to the anonymized text file.
|
|
124
|
+
- `MAPPING_FILE`: Path to the JSON mapping file.
|
|
125
|
+
|
|
126
|
+
**Example**:
|
|
127
|
+
```bash
|
|
128
|
+
pdf-anonymizer deanonymize \
|
|
129
|
+
data/anonymized/document.anonymized.md \
|
|
130
|
+
data/mappings/document.mapping.json
|
|
131
|
+
```
|
|
132
|
+
|
|
133
|
+
This will create a deanonymized version of the file at `data/deanonymized/document.deanonymized.md`.
|
|
134
|
+
|
|
135
|
+
---
|
|
136
|
+
|
|
137
|
+
## See Also
|
|
138
|
+
|
|
139
|
+
- **[Main Documentation](https://leo-gan.github.io/anonymizer/)** — Full guides including the 101 course.
|
|
140
|
+
- **[Core Package README](../pdf-anonymizer-core/README.md)** — Details on the underlying library.
|
|
141
|
+
- **[Recipes & Common Workflows](https://leo-gan.github.io/anonymizer/project/recipes/)** — Practical CLI usage examples (profiles, batching, external workflows, etc.).
|
|
142
|
+
- **[CLI Reference (full)](https://leo-gan.github.io/anonymizer/project/cli-usage/)** — Complete command reference on the docs site.
|
|
143
|
+
- **[Troubleshooting](https://leo-gan.github.io/anonymizer/project/troubleshooting/)** — Help with common CLI problems.
|
|
144
|
+
- **[Architecture](https://leo-gan.github.io/anonymizer/project/architecture/)** — How the CLI and core work together.
|
|
@@ -0,0 +1,131 @@
|
|
|
1
|
+
# 🦉🫥 PDF Anonymizer CLI
|
|
2
|
+
|
|
3
|
+
A command-line interface for anonymizing PDF, Markdown, and plain text files using LLMs.
|
|
4
|
+
|
|
5
|
+
- **High-Quality Anonymization**: Leverages LLMs to identify and replace Personally Identifiable Information (PII) with high accuracy.
|
|
6
|
+
- **Large File Support**: Consistently anonymizes large files (tested up to 1GB).
|
|
7
|
+
- **Multi-Provider & Cost-Effective**: Free to use with local [Ollama](https://ollama.com/) models. It also supports major providers like [OpenAI](https://openai.com/), [Anthropic](https://www.anthropic.com/), [Google](https://ai.google.com/), [Hugging Face](https://huggingface.co/), and [OpenRouter](https://openrouter.ai/).
|
|
8
|
+
- **Reversible**: Supports deanonymization to recover original data when needed.
|
|
9
|
+
- **Multi-Format**: Works with PDF, Markdown, and plain text files.
|
|
10
|
+
|
|
11
|
+
|
|
12
|
+
## Installation
|
|
13
|
+
|
|
14
|
+
Install the CLI with your favorite package manager. To use a specific LLM provider, you must install the corresponding extra.
|
|
15
|
+
|
|
16
|
+
- **Google**: `pip install "pdf-anonymizer-cli[google]"`
|
|
17
|
+
- **Ollama**: `pip install "pdf-anonymizer-cli[ollama]"`
|
|
18
|
+
- **Hugging Face**: `pip install "pdf-anonymizer-cli[huggingface]"`
|
|
19
|
+
- **OpenRouter**: `pip install "pdf-anonymizer-cli[openrouter]"`
|
|
20
|
+
- **OpenAI**: `pip install "pdf-anonymizer-cli[openai]"`
|
|
21
|
+
- **Anthropic**: `pip install "pdf-anonymizer-cli[anthropic]"`
|
|
22
|
+
|
|
23
|
+
You can also install multiple extras at once:
|
|
24
|
+
|
|
25
|
+
```bash
|
|
26
|
+
pip install "pdf-anonymizer-cli[google,openrouter]"
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
This installs the `pdf-anonymizer` executable.
|
|
30
|
+
|
|
31
|
+
|
|
32
|
+
## Environment Variables
|
|
33
|
+
|
|
34
|
+
The CLI will automatically load a `.env` file from the current directory or any parent directory. For consistency, it's recommended to place a single `.env` file at the root of the repository.
|
|
35
|
+
|
|
36
|
+
- `GOOGLE_API_KEY`: Required when using Google models.
|
|
37
|
+
- `HUGGING_FACE_TOKEN`: Required when using Hugging Face models. You can get a token from [here](https://huggingface.co/docs/hub/security-tokens).
|
|
38
|
+
- `OPENROUTER_API_KEY`: Required when using OpenRouter models.
|
|
39
|
+
- `OPENAI_API_KEY`: Required when using OpenAI models.
|
|
40
|
+
- `ANTHROPIC_API_KEY`: Required when using Anthropic models.
|
|
41
|
+
- `OLLAMA_HOST`: Optional, defaults to `http://localhost:11434` when using Ollama models.
|
|
42
|
+
|
|
43
|
+
Example `.env` file:
|
|
44
|
+
```env
|
|
45
|
+
GOOGLE_API_KEY="YOUR_API_KEY_HERE"
|
|
46
|
+
HUGGING_FACE_TOKEN="YOUR_HF_TOKEN_HERE"
|
|
47
|
+
OPENROUTER_API_KEY="YOUR_OPENROUTER_KEY"
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
## Usage
|
|
51
|
+
|
|
52
|
+
### Anonymize
|
|
53
|
+
|
|
54
|
+
The `run` command anonymizes one or more files.
|
|
55
|
+
|
|
56
|
+
```bash
|
|
57
|
+
pdf-anonymizer run FILE_PATH [FILE_PATH ...] \
|
|
58
|
+
[--characters-to-anonymize INTEGER] \
|
|
59
|
+
[--prompt-name {simple|detailed}] \
|
|
60
|
+
[--model-name TEXT] \
|
|
61
|
+
[--anonymized-entities PATH]
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
**Arguments**:
|
|
65
|
+
- `FILE_PATH`: Path to one or several PDF, Markdown, or text files for anonymization.
|
|
66
|
+
|
|
67
|
+
**Options**:
|
|
68
|
+
- `--characters-to-anonymize INTEGER`: Number of characters to process in each chunk (default: `100000`).
|
|
69
|
+
- `--prompt-name [simple|detailed]`: The prompt template to use (default: `detailed`).
|
|
70
|
+
- `--model-name TEXT`: The language model to use.
|
|
71
|
+
- `--anonymized-entities PATH`: Path to a file with a list of entities to anonymize.
|
|
72
|
+
|
|
73
|
+
**Models**:
|
|
74
|
+
You can use any of the predefined models below, or specify a new model using the format `"provider/model-name"`.
|
|
75
|
+
For example: `--model-name "google/gemini-flash-latest"`.
|
|
76
|
+
|
|
77
|
+
- **Google**: `gemini-2.5-pro`, `gemini-2.5-flash` (default), `gemini-2.5-flash-lite`.
|
|
78
|
+
- **Ollama**: `gemma:7b`, `phi4-mini`.
|
|
79
|
+
- **Hugging Face**: `openai/gpt-oss-20b`, `mistralai/Mistral-7B-Instruct-v0.1`, `HuggingFaceH4/zephyr-7b-beta`.
|
|
80
|
+
- **OpenRouter**: `openai/gpt-4o`, `google/gemini-pro`.
|
|
81
|
+
- **OpenAI**: `gpt-4o`, `gpt-5`.
|
|
82
|
+
- **Anthropic**: `claude-4-sonet`, `claude-4.5-sonet`.
|
|
83
|
+
|
|
84
|
+
### Examples
|
|
85
|
+
|
|
86
|
+
**Basic anonymization with the default model (Google)**:
|
|
87
|
+
```bash
|
|
88
|
+
pdf-anonymizer run document.pdf
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
**A new model (Google) and a simple prompt**:
|
|
92
|
+
```bash
|
|
93
|
+
pdf-anonymizer run notes.md --model-name "google/gemini-flash-latest" --prompt-name simple
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
**Using an OpenRouter model**:
|
|
97
|
+
```bash
|
|
98
|
+
pdf-anonymizer run report.pdf --model-name "openai/gpt-4o"
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
### Deanonymize
|
|
102
|
+
|
|
103
|
+
The `deanonymize` command reverts anonymization using a mapping file.
|
|
104
|
+
|
|
105
|
+
```bash
|
|
106
|
+
pdf-anonymizer deanonymize ANONYMIZED_FILE MAPPING_FILE
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
**Arguments**:
|
|
110
|
+
- `ANONYMIZED_FILE`: Path to the anonymized text file.
|
|
111
|
+
- `MAPPING_FILE`: Path to the JSON mapping file.
|
|
112
|
+
|
|
113
|
+
**Example**:
|
|
114
|
+
```bash
|
|
115
|
+
pdf-anonymizer deanonymize \
|
|
116
|
+
data/anonymized/document.anonymized.md \
|
|
117
|
+
data/mappings/document.mapping.json
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
This will create a deanonymized version of the file at `data/deanonymized/document.deanonymized.md`.
|
|
121
|
+
|
|
122
|
+
---
|
|
123
|
+
|
|
124
|
+
## See Also
|
|
125
|
+
|
|
126
|
+
- **[Main Documentation](https://leo-gan.github.io/anonymizer/)** — Full guides including the 101 course.
|
|
127
|
+
- **[Core Package README](../pdf-anonymizer-core/README.md)** — Details on the underlying library.
|
|
128
|
+
- **[Recipes & Common Workflows](https://leo-gan.github.io/anonymizer/project/recipes/)** — Practical CLI usage examples (profiles, batching, external workflows, etc.).
|
|
129
|
+
- **[CLI Reference (full)](https://leo-gan.github.io/anonymizer/project/cli-usage/)** — Complete command reference on the docs site.
|
|
130
|
+
- **[Troubleshooting](https://leo-gan.github.io/anonymizer/project/troubleshooting/)** — Help with common CLI problems.
|
|
131
|
+
- **[Architecture](https://leo-gan.github.io/anonymizer/project/architecture/)** — How the CLI and core work together.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
[project]
|
|
2
2
|
name = "pdf-anonymizer-cli"
|
|
3
|
-
version = "0.3.
|
|
3
|
+
version = "0.3.3"
|
|
4
4
|
description = "CLI for a tool to anonymize PDF, Markdown, and plain text files using LLMs."
|
|
5
5
|
authors = [{ name = "Leonid Ganeline", email = "leo.gan.57@gmail.com" }]
|
|
6
6
|
license = { text = "MIT" }
|
|
@@ -7,15 +7,14 @@ from typing import Dict, List, Optional
|
|
|
7
7
|
import typer
|
|
8
8
|
from dotenv import load_dotenv
|
|
9
9
|
from pdf_anonymizer_core.conf import (
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
DEFAULT_PROMPT_NAME,
|
|
13
|
-
ModelName,
|
|
14
|
-
ModelProvider,
|
|
10
|
+
DEFAULT_LOG_FILE,
|
|
11
|
+
ConfigProfile,
|
|
15
12
|
PromptEnum,
|
|
16
|
-
|
|
13
|
+
get_config_for_profile,
|
|
14
|
+
get_provider_and_model_name,
|
|
17
15
|
)
|
|
18
16
|
from pdf_anonymizer_core.core import anonymize_file
|
|
17
|
+
from pdf_anonymizer_core.llm_provider import configure_cache
|
|
19
18
|
from pdf_anonymizer_core.prompts import detailed, simple
|
|
20
19
|
from pdf_anonymizer_core.utils import (
|
|
21
20
|
consolidate_mapping,
|
|
@@ -24,10 +23,11 @@ from pdf_anonymizer_core.utils import (
|
|
|
24
23
|
)
|
|
25
24
|
from typing_extensions import Annotated
|
|
26
25
|
|
|
26
|
+
# Setup logging using log file configured in conf.py
|
|
27
27
|
logging.basicConfig(
|
|
28
28
|
level=logging.INFO,
|
|
29
29
|
format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
|
|
30
|
-
handlers=[logging.FileHandler(
|
|
30
|
+
handlers=[logging.FileHandler(DEFAULT_LOG_FILE), logging.StreamHandler()],
|
|
31
31
|
)
|
|
32
32
|
|
|
33
33
|
app = typer.Typer()
|
|
@@ -54,24 +54,34 @@ def run(
|
|
|
54
54
|
resolve_path=True,
|
|
55
55
|
),
|
|
56
56
|
],
|
|
57
|
+
config_profile: Annotated[
|
|
58
|
+
ConfigProfile,
|
|
59
|
+
typer.Option(
|
|
60
|
+
"--config-profile",
|
|
61
|
+
"-p",
|
|
62
|
+
help="The configuration profile to use (best-quality, best-speed, best-cost).",
|
|
63
|
+
case_sensitive=False,
|
|
64
|
+
),
|
|
65
|
+
] = ConfigProfile.BEST_SPEED,
|
|
57
66
|
characters_to_anonymize: Annotated[
|
|
58
|
-
int,
|
|
59
|
-
typer.Option(
|
|
60
|
-
|
|
67
|
+
Optional[int],
|
|
68
|
+
typer.Option(
|
|
69
|
+
help="Override number of characters to send for anonymization in one go (chunk size)."
|
|
70
|
+
),
|
|
71
|
+
] = None,
|
|
61
72
|
prompt_name: Annotated[
|
|
62
|
-
PromptEnum,
|
|
73
|
+
Optional[PromptEnum],
|
|
63
74
|
typer.Option(
|
|
64
|
-
help="
|
|
75
|
+
help="Override prompt template to use for anonymization.",
|
|
65
76
|
case_sensitive=False,
|
|
66
77
|
),
|
|
67
|
-
] =
|
|
78
|
+
] = None,
|
|
68
79
|
model_name: Annotated[
|
|
69
|
-
|
|
80
|
+
Optional[str],
|
|
70
81
|
typer.Option(
|
|
71
|
-
help="
|
|
72
|
-
case_sensitive=False,
|
|
82
|
+
help="Override the language model to use for anonymization.",
|
|
73
83
|
),
|
|
74
|
-
] =
|
|
84
|
+
] = None,
|
|
75
85
|
anonymized_entities: Annotated[
|
|
76
86
|
Optional[Path],
|
|
77
87
|
typer.Option(
|
|
@@ -88,34 +98,45 @@ def run(
|
|
|
88
98
|
) -> None:
|
|
89
99
|
"""
|
|
90
100
|
Anonymize one or more files by replacing PII with anonymized placeholders.
|
|
91
|
-
|
|
92
|
-
Args:
|
|
93
|
-
file_paths: List of paths to files to process.
|
|
94
|
-
characters_to_anonymize: Number of characters to process in each chunk.
|
|
95
|
-
prompt_name: The prompt template to use for anonymization.
|
|
96
|
-
model_name: The language model to use for anonymization.
|
|
97
|
-
anonymized_entities: A file with a list of entities to anonymize.
|
|
98
101
|
"""
|
|
99
102
|
load_environment()
|
|
100
103
|
|
|
101
|
-
|
|
102
|
-
|
|
104
|
+
# Get configuration based on profile and optional user overrides
|
|
105
|
+
config = get_config_for_profile(
|
|
106
|
+
profile=config_profile,
|
|
107
|
+
model_name=model_name,
|
|
108
|
+
prompt_name=prompt_name,
|
|
109
|
+
chunk_size=characters_to_anonymize,
|
|
110
|
+
)
|
|
111
|
+
|
|
112
|
+
# Configure LLM caching with values from configuration
|
|
113
|
+
configure_cache(
|
|
114
|
+
enabled=config.enable_cache,
|
|
115
|
+
cache_dir=config.cache_dir,
|
|
116
|
+
cache_file=config.cache_file,
|
|
117
|
+
)
|
|
118
|
+
|
|
119
|
+
provider_name, _ = get_provider_and_model_name(config.model_name)
|
|
120
|
+
if provider_name == "google":
|
|
121
|
+
if "gemini" in config.model_name and not os.getenv("GOOGLE_API_KEY"):
|
|
103
122
|
logging.error(
|
|
104
123
|
"Error: GOOGLE_API_KEY not found. Please set it in the .env file."
|
|
105
124
|
)
|
|
106
125
|
sys.exit(1)
|
|
107
126
|
|
|
127
|
+
logging.info(f"Using configuration profile: {config_profile.value}")
|
|
108
128
|
logging.info(f" --file-paths: {file_paths}")
|
|
109
|
-
logging.info(f" --
|
|
110
|
-
logging.info(f" --
|
|
129
|
+
logging.info(f" --chunk-size: {config.chunk_size}")
|
|
130
|
+
logging.info(f" --chunk-overlap: {config.chunk_overlap}")
|
|
131
|
+
logging.info(f" --model-name: {config.model_name}")
|
|
111
132
|
|
|
112
133
|
# Select the appropriate prompt template
|
|
113
134
|
prompt_templates: Dict[str, str] = {
|
|
114
|
-
PromptEnum.simple: simple.prompt_template,
|
|
115
|
-
PromptEnum.detailed: detailed.prompt_template,
|
|
135
|
+
PromptEnum.simple.value: simple.prompt_template,
|
|
136
|
+
PromptEnum.detailed.value: detailed.prompt_template,
|
|
116
137
|
}
|
|
117
|
-
prompt_template: str = prompt_templates[prompt_name]
|
|
118
|
-
logging.info(f" --prompt-name: {prompt_name
|
|
138
|
+
prompt_template: str = prompt_templates[config.prompt_name]
|
|
139
|
+
logging.info(f" --prompt-name: {config.prompt_name}")
|
|
119
140
|
|
|
120
141
|
entities_to_anonymize = None
|
|
121
142
|
if anonymized_entities:
|
|
@@ -129,11 +150,16 @@ def run(
|
|
|
129
150
|
logging.info("=" * 40)
|
|
130
151
|
logging.info(f"Processing file {i}/{len(file_paths)}: {file_path}")
|
|
131
152
|
full_anonymized_text, final_mapping = anonymize_file(
|
|
132
|
-
str(file_path),
|
|
133
|
-
characters_to_anonymize,
|
|
134
|
-
prompt_template,
|
|
135
|
-
model_name.
|
|
136
|
-
entities_to_anonymize,
|
|
153
|
+
file_path=str(file_path),
|
|
154
|
+
characters_to_anonymize=config.chunk_size,
|
|
155
|
+
prompt_template=prompt_template,
|
|
156
|
+
model_name=config.model_name,
|
|
157
|
+
anonymized_entities=entities_to_anonymize,
|
|
158
|
+
chunk_overlap=config.chunk_overlap,
|
|
159
|
+
regex_patterns=config.regex_patterns,
|
|
160
|
+
max_retries=config.max_retries,
|
|
161
|
+
base_retry_delay=config.base_retry_delay,
|
|
162
|
+
max_retry_delay=config.max_retry_delay,
|
|
137
163
|
)
|
|
138
164
|
|
|
139
165
|
if full_anonymized_text and final_mapping:
|
|
@@ -183,10 +209,6 @@ def deanonymize(
|
|
|
183
209
|
) -> None:
|
|
184
210
|
"""
|
|
185
211
|
Deanonymize a file using a mapping file.
|
|
186
|
-
|
|
187
|
-
Args:
|
|
188
|
-
anonymized_file: Path to the anonymized file.
|
|
189
|
-
mapping_file: Path to the mapping file.
|
|
190
212
|
"""
|
|
191
213
|
logging.info(f"Deanonymizing '{anonymized_file}' using '{mapping_file}'")
|
|
192
214
|
deanonymized_output_file, stats_file = deanonymize_file(
|
|
@@ -0,0 +1,144 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: pdf-anonymizer-cli
|
|
3
|
+
Version: 0.3.3
|
|
4
|
+
Summary: CLI for a tool to anonymize PDF, Markdown, and plain text files using LLMs.
|
|
5
|
+
Author-email: Leonid Ganeline <leo.gan.57@gmail.com>
|
|
6
|
+
License: MIT
|
|
7
|
+
Project-URL: repository, https://github.com/leo-gan/anonymizer
|
|
8
|
+
Requires-Python: >=3.10
|
|
9
|
+
Description-Content-Type: text/markdown
|
|
10
|
+
Requires-Dist: typer
|
|
11
|
+
Requires-Dist: rich
|
|
12
|
+
Requires-Dist: pdf-anonymizer-core
|
|
13
|
+
|
|
14
|
+
# 🦉🫥 PDF Anonymizer CLI
|
|
15
|
+
|
|
16
|
+
A command-line interface for anonymizing PDF, Markdown, and plain text files using LLMs.
|
|
17
|
+
|
|
18
|
+
- **High-Quality Anonymization**: Leverages LLMs to identify and replace Personally Identifiable Information (PII) with high accuracy.
|
|
19
|
+
- **Large File Support**: Consistently anonymizes large files (tested up to 1GB).
|
|
20
|
+
- **Multi-Provider & Cost-Effective**: Free to use with local [Ollama](https://ollama.com/) models. It also supports major providers like [OpenAI](https://openai.com/), [Anthropic](https://www.anthropic.com/), [Google](https://ai.google.com/), [Hugging Face](https://huggingface.co/), and [OpenRouter](https://openrouter.ai/).
|
|
21
|
+
- **Reversible**: Supports deanonymization to recover original data when needed.
|
|
22
|
+
- **Multi-Format**: Works with PDF, Markdown, and plain text files.
|
|
23
|
+
|
|
24
|
+
|
|
25
|
+
## Installation
|
|
26
|
+
|
|
27
|
+
Install the CLI with your favorite package manager. To use a specific LLM provider, you must install the corresponding extra.
|
|
28
|
+
|
|
29
|
+
- **Google**: `pip install "pdf-anonymizer-cli[google]"`
|
|
30
|
+
- **Ollama**: `pip install "pdf-anonymizer-cli[ollama]"`
|
|
31
|
+
- **Hugging Face**: `pip install "pdf-anonymizer-cli[huggingface]"`
|
|
32
|
+
- **OpenRouter**: `pip install "pdf-anonymizer-cli[openrouter]"`
|
|
33
|
+
- **OpenAI**: `pip install "pdf-anonymizer-cli[openai]"`
|
|
34
|
+
- **Anthropic**: `pip install "pdf-anonymizer-cli[anthropic]"`
|
|
35
|
+
|
|
36
|
+
You can also install multiple extras at once:
|
|
37
|
+
|
|
38
|
+
```bash
|
|
39
|
+
pip install "pdf-anonymizer-cli[google,openrouter]"
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
This installs the `pdf-anonymizer` executable.
|
|
43
|
+
|
|
44
|
+
|
|
45
|
+
## Environment Variables
|
|
46
|
+
|
|
47
|
+
The CLI will automatically load a `.env` file from the current directory or any parent directory. For consistency, it's recommended to place a single `.env` file at the root of the repository.
|
|
48
|
+
|
|
49
|
+
- `GOOGLE_API_KEY`: Required when using Google models.
|
|
50
|
+
- `HUGGING_FACE_TOKEN`: Required when using Hugging Face models. You can get a token from [here](https://huggingface.co/docs/hub/security-tokens).
|
|
51
|
+
- `OPENROUTER_API_KEY`: Required when using OpenRouter models.
|
|
52
|
+
- `OPENAI_API_KEY`: Required when using OpenAI models.
|
|
53
|
+
- `ANTHROPIC_API_KEY`: Required when using Anthropic models.
|
|
54
|
+
- `OLLAMA_HOST`: Optional, defaults to `http://localhost:11434` when using Ollama models.
|
|
55
|
+
|
|
56
|
+
Example `.env` file:
|
|
57
|
+
```env
|
|
58
|
+
GOOGLE_API_KEY="YOUR_API_KEY_HERE"
|
|
59
|
+
HUGGING_FACE_TOKEN="YOUR_HF_TOKEN_HERE"
|
|
60
|
+
OPENROUTER_API_KEY="YOUR_OPENROUTER_KEY"
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
## Usage
|
|
64
|
+
|
|
65
|
+
### Anonymize
|
|
66
|
+
|
|
67
|
+
The `run` command anonymizes one or more files.
|
|
68
|
+
|
|
69
|
+
```bash
|
|
70
|
+
pdf-anonymizer run FILE_PATH [FILE_PATH ...] \
|
|
71
|
+
[--characters-to-anonymize INTEGER] \
|
|
72
|
+
[--prompt-name {simple|detailed}] \
|
|
73
|
+
[--model-name TEXT] \
|
|
74
|
+
[--anonymized-entities PATH]
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
**Arguments**:
|
|
78
|
+
- `FILE_PATH`: Path to one or several PDF, Markdown, or text files for anonymization.
|
|
79
|
+
|
|
80
|
+
**Options**:
|
|
81
|
+
- `--characters-to-anonymize INTEGER`: Number of characters to process in each chunk (default: `100000`).
|
|
82
|
+
- `--prompt-name [simple|detailed]`: The prompt template to use (default: `detailed`).
|
|
83
|
+
- `--model-name TEXT`: The language model to use.
|
|
84
|
+
- `--anonymized-entities PATH`: Path to a file with a list of entities to anonymize.
|
|
85
|
+
|
|
86
|
+
**Models**:
|
|
87
|
+
You can use any of the predefined models below, or specify a new model using the format `"provider/model-name"`.
|
|
88
|
+
For example: `--model-name "google/gemini-flash-latest"`.
|
|
89
|
+
|
|
90
|
+
- **Google**: `gemini-2.5-pro`, `gemini-2.5-flash` (default), `gemini-2.5-flash-lite`.
|
|
91
|
+
- **Ollama**: `gemma:7b`, `phi4-mini`.
|
|
92
|
+
- **Hugging Face**: `openai/gpt-oss-20b`, `mistralai/Mistral-7B-Instruct-v0.1`, `HuggingFaceH4/zephyr-7b-beta`.
|
|
93
|
+
- **OpenRouter**: `openai/gpt-4o`, `google/gemini-pro`.
|
|
94
|
+
- **OpenAI**: `gpt-4o`, `gpt-5`.
|
|
95
|
+
- **Anthropic**: `claude-4-sonet`, `claude-4.5-sonet`.
|
|
96
|
+
|
|
97
|
+
### Examples
|
|
98
|
+
|
|
99
|
+
**Basic anonymization with the default model (Google)**:
|
|
100
|
+
```bash
|
|
101
|
+
pdf-anonymizer run document.pdf
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
**A new model (Google) and a simple prompt**:
|
|
105
|
+
```bash
|
|
106
|
+
pdf-anonymizer run notes.md --model-name "google/gemini-flash-latest" --prompt-name simple
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
**Using an OpenRouter model**:
|
|
110
|
+
```bash
|
|
111
|
+
pdf-anonymizer run report.pdf --model-name "openai/gpt-4o"
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
### Deanonymize
|
|
115
|
+
|
|
116
|
+
The `deanonymize` command reverts anonymization using a mapping file.
|
|
117
|
+
|
|
118
|
+
```bash
|
|
119
|
+
pdf-anonymizer deanonymize ANONYMIZED_FILE MAPPING_FILE
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
**Arguments**:
|
|
123
|
+
- `ANONYMIZED_FILE`: Path to the anonymized text file.
|
|
124
|
+
- `MAPPING_FILE`: Path to the JSON mapping file.
|
|
125
|
+
|
|
126
|
+
**Example**:
|
|
127
|
+
```bash
|
|
128
|
+
pdf-anonymizer deanonymize \
|
|
129
|
+
data/anonymized/document.anonymized.md \
|
|
130
|
+
data/mappings/document.mapping.json
|
|
131
|
+
```
|
|
132
|
+
|
|
133
|
+
This will create a deanonymized version of the file at `data/deanonymized/document.deanonymized.md`.
|
|
134
|
+
|
|
135
|
+
---
|
|
136
|
+
|
|
137
|
+
## See Also
|
|
138
|
+
|
|
139
|
+
- **[Main Documentation](https://leo-gan.github.io/anonymizer/)** — Full guides including the 101 course.
|
|
140
|
+
- **[Core Package README](../pdf-anonymizer-core/README.md)** — Details on the underlying library.
|
|
141
|
+
- **[Recipes & Common Workflows](https://leo-gan.github.io/anonymizer/project/recipes/)** — Practical CLI usage examples (profiles, batching, external workflows, etc.).
|
|
142
|
+
- **[CLI Reference (full)](https://leo-gan.github.io/anonymizer/project/cli-usage/)** — Complete command reference on the docs site.
|
|
143
|
+
- **[Troubleshooting](https://leo-gan.github.io/anonymizer/project/troubleshooting/)** — Help with common CLI problems.
|
|
144
|
+
- **[Architecture](https://leo-gan.github.io/anonymizer/project/architecture/)** — How the CLI and core work together.
|
|
@@ -1,103 +0,0 @@
|
|
|
1
|
-
Metadata-Version: 2.4
|
|
2
|
-
Name: pdf-anonymizer-cli
|
|
3
|
-
Version: 0.3.1
|
|
4
|
-
Summary: CLI for a tool to anonymize PDF, Markdown, and plain text files using LLMs.
|
|
5
|
-
Author-email: Leonid Ganeline <leo.gan.57@gmail.com>
|
|
6
|
-
License: MIT
|
|
7
|
-
Project-URL: repository, https://github.com/leo-gan/anonymizer
|
|
8
|
-
Requires-Python: >=3.10
|
|
9
|
-
Description-Content-Type: text/markdown
|
|
10
|
-
Requires-Dist: typer
|
|
11
|
-
Requires-Dist: rich
|
|
12
|
-
Requires-Dist: pdf-anonymizer-core
|
|
13
|
-
|
|
14
|
-
# 🦉🫥 PDF Anonymizer CLI
|
|
15
|
-
|
|
16
|
-
A command-line interface for anonymizing PDF, Markdown, and plain text files using LLMs.
|
|
17
|
-
|
|
18
|
-
## Installation
|
|
19
|
-
|
|
20
|
-
This project uses `uv` and is structured as a monorepo. The dependencies for the CLI and its core library are managed at the root of the project.
|
|
21
|
-
|
|
22
|
-
1. **Install `uv`**: Follow the [official installation instructions](https://astral.sh/docs/uv#installation).
|
|
23
|
-
2. **Install dependencies from the repository root**:
|
|
24
|
-
```bash
|
|
25
|
-
# From the repository root
|
|
26
|
-
uv sync
|
|
27
|
-
```
|
|
28
|
-
This installs the `pdf-anonymizer` executable.
|
|
29
|
-
|
|
30
|
-
## Environment Variables
|
|
31
|
-
|
|
32
|
-
The CLI will automatically load a `.env` file from the current directory or any parent directory. For consistency, it's recommended to place a single `.env` file at the root of the repository.
|
|
33
|
-
|
|
34
|
-
- `GOOGLE_API_KEY`: Required when using Google's Gemini models.
|
|
35
|
-
- `OLLAMA_HOST`: Optional, defaults to `http://localhost:11434` when using local Ollama models.
|
|
36
|
-
- `HUGGING_FACE_TOKEN`: Required when using Hugging Face models. You can get a token from [here](https://huggingface.co/docs/hub/security-tokens).
|
|
37
|
-
|
|
38
|
-
Example `.env` file:
|
|
39
|
-
```env
|
|
40
|
-
GOOGLE_API_KEY="YOUR_API_KEY_HERE"
|
|
41
|
-
HUGGING_FACE_TOKEN="YOUR_HF_TOKEN_HERE"
|
|
42
|
-
```
|
|
43
|
-
|
|
44
|
-
## Usage
|
|
45
|
-
|
|
46
|
-
### Anonymize
|
|
47
|
-
|
|
48
|
-
The `run` command anonymizes one or more files.
|
|
49
|
-
|
|
50
|
-
```bash
|
|
51
|
-
pdf-anonymizer run FILE_PATH [FILE_PATH ...] \
|
|
52
|
-
[--characters-to-anonymize INTEGER] \
|
|
53
|
-
[--prompt-name {simple|detailed}] \
|
|
54
|
-
[--model-name TEXT] \
|
|
55
|
-
[--anonymized-entities PATH]
|
|
56
|
-
```
|
|
57
|
-
|
|
58
|
-
**Arguments**:
|
|
59
|
-
- `FILE_PATH`: Path to one or several PDF, Markdown, or text files for anonymization.
|
|
60
|
-
|
|
61
|
-
**Options**:
|
|
62
|
-
- `--characters-to-anonymize INTEGER`: Number of characters to process in each chunk (default: `100000`).
|
|
63
|
-
- `--prompt-name [simple|detailed]`: The prompt template to use (default: `detailed`).
|
|
64
|
-
- `--model-name TEXT`: The language model to use.
|
|
65
|
-
- `--anonymized-entities PATH`: Path to a file with a list of entities to anonymize.
|
|
66
|
-
|
|
67
|
-
**Models**:
|
|
68
|
-
- **Google**: `gemini-2.5-pro`, `gemini-2.5-flash` (default), `gemini-2.5-flash-lite`.
|
|
69
|
-
- **Ollama**: `gemma:7b`, `phi4-mini`.
|
|
70
|
-
- **Hugging Face**: `openai/gpt-oss-20b`, `mistralai/Mistral-7B-Instruct-v0.1`, `HuggingFaceH4/zephyr-7b-beta`.
|
|
71
|
-
|
|
72
|
-
### Examples
|
|
73
|
-
|
|
74
|
-
**Basic anonymization**:
|
|
75
|
-
```bash
|
|
76
|
-
pdf-anonymizer run document.pdf
|
|
77
|
-
```
|
|
78
|
-
|
|
79
|
-
**Custom model and prompt**:
|
|
80
|
-
```bash
|
|
81
|
-
pdf-anonymizer run notes.md --model-name phi4-mini --prompt-name simple
|
|
82
|
-
```
|
|
83
|
-
|
|
84
|
-
### Deanonymize
|
|
85
|
-
|
|
86
|
-
The `deanonymize` command reverts anonymization using a mapping file.
|
|
87
|
-
|
|
88
|
-
```bash
|
|
89
|
-
pdf-anonymizer deanonymize ANONYMIZED_FILE MAPPING_FILE
|
|
90
|
-
```
|
|
91
|
-
|
|
92
|
-
**Arguments**:
|
|
93
|
-
- `ANONYMIZED_FILE`: Path to the anonymized text file.
|
|
94
|
-
- `MAPPING_FILE`: Path to the JSON mapping file.
|
|
95
|
-
|
|
96
|
-
**Example**:
|
|
97
|
-
```bash
|
|
98
|
-
pdf-anonymizer deanonymize \
|
|
99
|
-
data/anonymized/document.anonymized.md \
|
|
100
|
-
data/mappings/document.mapping.json
|
|
101
|
-
```
|
|
102
|
-
|
|
103
|
-
This will create a deanonymized version of the file at `data/deanonymized/document.deanonymized.md`.
|
|
@@ -1,90 +0,0 @@
|
|
|
1
|
-
# 🦉🫥 PDF Anonymizer CLI
|
|
2
|
-
|
|
3
|
-
A command-line interface for anonymizing PDF, Markdown, and plain text files using LLMs.
|
|
4
|
-
|
|
5
|
-
## Installation
|
|
6
|
-
|
|
7
|
-
This project uses `uv` and is structured as a monorepo. The dependencies for the CLI and its core library are managed at the root of the project.
|
|
8
|
-
|
|
9
|
-
1. **Install `uv`**: Follow the [official installation instructions](https://astral.sh/docs/uv#installation).
|
|
10
|
-
2. **Install dependencies from the repository root**:
|
|
11
|
-
```bash
|
|
12
|
-
# From the repository root
|
|
13
|
-
uv sync
|
|
14
|
-
```
|
|
15
|
-
This installs the `pdf-anonymizer` executable.
|
|
16
|
-
|
|
17
|
-
## Environment Variables
|
|
18
|
-
|
|
19
|
-
The CLI will automatically load a `.env` file from the current directory or any parent directory. For consistency, it's recommended to place a single `.env` file at the root of the repository.
|
|
20
|
-
|
|
21
|
-
- `GOOGLE_API_KEY`: Required when using Google's Gemini models.
|
|
22
|
-
- `OLLAMA_HOST`: Optional, defaults to `http://localhost:11434` when using local Ollama models.
|
|
23
|
-
- `HUGGING_FACE_TOKEN`: Required when using Hugging Face models. You can get a token from [here](https://huggingface.co/docs/hub/security-tokens).
|
|
24
|
-
|
|
25
|
-
Example `.env` file:
|
|
26
|
-
```env
|
|
27
|
-
GOOGLE_API_KEY="YOUR_API_KEY_HERE"
|
|
28
|
-
HUGGING_FACE_TOKEN="YOUR_HF_TOKEN_HERE"
|
|
29
|
-
```
|
|
30
|
-
|
|
31
|
-
## Usage
|
|
32
|
-
|
|
33
|
-
### Anonymize
|
|
34
|
-
|
|
35
|
-
The `run` command anonymizes one or more files.
|
|
36
|
-
|
|
37
|
-
```bash
|
|
38
|
-
pdf-anonymizer run FILE_PATH [FILE_PATH ...] \
|
|
39
|
-
[--characters-to-anonymize INTEGER] \
|
|
40
|
-
[--prompt-name {simple|detailed}] \
|
|
41
|
-
[--model-name TEXT] \
|
|
42
|
-
[--anonymized-entities PATH]
|
|
43
|
-
```
|
|
44
|
-
|
|
45
|
-
**Arguments**:
|
|
46
|
-
- `FILE_PATH`: Path to one or several PDF, Markdown, or text files for anonymization.
|
|
47
|
-
|
|
48
|
-
**Options**:
|
|
49
|
-
- `--characters-to-anonymize INTEGER`: Number of characters to process in each chunk (default: `100000`).
|
|
50
|
-
- `--prompt-name [simple|detailed]`: The prompt template to use (default: `detailed`).
|
|
51
|
-
- `--model-name TEXT`: The language model to use.
|
|
52
|
-
- `--anonymized-entities PATH`: Path to a file with a list of entities to anonymize.
|
|
53
|
-
|
|
54
|
-
**Models**:
|
|
55
|
-
- **Google**: `gemini-2.5-pro`, `gemini-2.5-flash` (default), `gemini-2.5-flash-lite`.
|
|
56
|
-
- **Ollama**: `gemma:7b`, `phi4-mini`.
|
|
57
|
-
- **Hugging Face**: `openai/gpt-oss-20b`, `mistralai/Mistral-7B-Instruct-v0.1`, `HuggingFaceH4/zephyr-7b-beta`.
|
|
58
|
-
|
|
59
|
-
### Examples
|
|
60
|
-
|
|
61
|
-
**Basic anonymization**:
|
|
62
|
-
```bash
|
|
63
|
-
pdf-anonymizer run document.pdf
|
|
64
|
-
```
|
|
65
|
-
|
|
66
|
-
**Custom model and prompt**:
|
|
67
|
-
```bash
|
|
68
|
-
pdf-anonymizer run notes.md --model-name phi4-mini --prompt-name simple
|
|
69
|
-
```
|
|
70
|
-
|
|
71
|
-
### Deanonymize
|
|
72
|
-
|
|
73
|
-
The `deanonymize` command reverts anonymization using a mapping file.
|
|
74
|
-
|
|
75
|
-
```bash
|
|
76
|
-
pdf-anonymizer deanonymize ANONYMIZED_FILE MAPPING_FILE
|
|
77
|
-
```
|
|
78
|
-
|
|
79
|
-
**Arguments**:
|
|
80
|
-
- `ANONYMIZED_FILE`: Path to the anonymized text file.
|
|
81
|
-
- `MAPPING_FILE`: Path to the JSON mapping file.
|
|
82
|
-
|
|
83
|
-
**Example**:
|
|
84
|
-
```bash
|
|
85
|
-
pdf-anonymizer deanonymize \
|
|
86
|
-
data/anonymized/document.anonymized.md \
|
|
87
|
-
data/mappings/document.mapping.json
|
|
88
|
-
```
|
|
89
|
-
|
|
90
|
-
This will create a deanonymized version of the file at `data/deanonymized/document.deanonymized.md`.
|
|
@@ -1,103 +0,0 @@
|
|
|
1
|
-
Metadata-Version: 2.4
|
|
2
|
-
Name: pdf-anonymizer-cli
|
|
3
|
-
Version: 0.3.1
|
|
4
|
-
Summary: CLI for a tool to anonymize PDF, Markdown, and plain text files using LLMs.
|
|
5
|
-
Author-email: Leonid Ganeline <leo.gan.57@gmail.com>
|
|
6
|
-
License: MIT
|
|
7
|
-
Project-URL: repository, https://github.com/leo-gan/anonymizer
|
|
8
|
-
Requires-Python: >=3.10
|
|
9
|
-
Description-Content-Type: text/markdown
|
|
10
|
-
Requires-Dist: typer
|
|
11
|
-
Requires-Dist: rich
|
|
12
|
-
Requires-Dist: pdf-anonymizer-core
|
|
13
|
-
|
|
14
|
-
# 🦉🫥 PDF Anonymizer CLI
|
|
15
|
-
|
|
16
|
-
A command-line interface for anonymizing PDF, Markdown, and plain text files using LLMs.
|
|
17
|
-
|
|
18
|
-
## Installation
|
|
19
|
-
|
|
20
|
-
This project uses `uv` and is structured as a monorepo. The dependencies for the CLI and its core library are managed at the root of the project.
|
|
21
|
-
|
|
22
|
-
1. **Install `uv`**: Follow the [official installation instructions](https://astral.sh/docs/uv#installation).
|
|
23
|
-
2. **Install dependencies from the repository root**:
|
|
24
|
-
```bash
|
|
25
|
-
# From the repository root
|
|
26
|
-
uv sync
|
|
27
|
-
```
|
|
28
|
-
This installs the `pdf-anonymizer` executable.
|
|
29
|
-
|
|
30
|
-
## Environment Variables
|
|
31
|
-
|
|
32
|
-
The CLI will automatically load a `.env` file from the current directory or any parent directory. For consistency, it's recommended to place a single `.env` file at the root of the repository.
|
|
33
|
-
|
|
34
|
-
- `GOOGLE_API_KEY`: Required when using Google's Gemini models.
|
|
35
|
-
- `OLLAMA_HOST`: Optional, defaults to `http://localhost:11434` when using local Ollama models.
|
|
36
|
-
- `HUGGING_FACE_TOKEN`: Required when using Hugging Face models. You can get a token from [here](https://huggingface.co/docs/hub/security-tokens).
|
|
37
|
-
|
|
38
|
-
Example `.env` file:
|
|
39
|
-
```env
|
|
40
|
-
GOOGLE_API_KEY="YOUR_API_KEY_HERE"
|
|
41
|
-
HUGGING_FACE_TOKEN="YOUR_HF_TOKEN_HERE"
|
|
42
|
-
```
|
|
43
|
-
|
|
44
|
-
## Usage
|
|
45
|
-
|
|
46
|
-
### Anonymize
|
|
47
|
-
|
|
48
|
-
The `run` command anonymizes one or more files.
|
|
49
|
-
|
|
50
|
-
```bash
|
|
51
|
-
pdf-anonymizer run FILE_PATH [FILE_PATH ...] \
|
|
52
|
-
[--characters-to-anonymize INTEGER] \
|
|
53
|
-
[--prompt-name {simple|detailed}] \
|
|
54
|
-
[--model-name TEXT] \
|
|
55
|
-
[--anonymized-entities PATH]
|
|
56
|
-
```
|
|
57
|
-
|
|
58
|
-
**Arguments**:
|
|
59
|
-
- `FILE_PATH`: Path to one or several PDF, Markdown, or text files for anonymization.
|
|
60
|
-
|
|
61
|
-
**Options**:
|
|
62
|
-
- `--characters-to-anonymize INTEGER`: Number of characters to process in each chunk (default: `100000`).
|
|
63
|
-
- `--prompt-name [simple|detailed]`: The prompt template to use (default: `detailed`).
|
|
64
|
-
- `--model-name TEXT`: The language model to use.
|
|
65
|
-
- `--anonymized-entities PATH`: Path to a file with a list of entities to anonymize.
|
|
66
|
-
|
|
67
|
-
**Models**:
|
|
68
|
-
- **Google**: `gemini-2.5-pro`, `gemini-2.5-flash` (default), `gemini-2.5-flash-lite`.
|
|
69
|
-
- **Ollama**: `gemma:7b`, `phi4-mini`.
|
|
70
|
-
- **Hugging Face**: `openai/gpt-oss-20b`, `mistralai/Mistral-7B-Instruct-v0.1`, `HuggingFaceH4/zephyr-7b-beta`.
|
|
71
|
-
|
|
72
|
-
### Examples
|
|
73
|
-
|
|
74
|
-
**Basic anonymization**:
|
|
75
|
-
```bash
|
|
76
|
-
pdf-anonymizer run document.pdf
|
|
77
|
-
```
|
|
78
|
-
|
|
79
|
-
**Custom model and prompt**:
|
|
80
|
-
```bash
|
|
81
|
-
pdf-anonymizer run notes.md --model-name phi4-mini --prompt-name simple
|
|
82
|
-
```
|
|
83
|
-
|
|
84
|
-
### Deanonymize
|
|
85
|
-
|
|
86
|
-
The `deanonymize` command reverts anonymization using a mapping file.
|
|
87
|
-
|
|
88
|
-
```bash
|
|
89
|
-
pdf-anonymizer deanonymize ANONYMIZED_FILE MAPPING_FILE
|
|
90
|
-
```
|
|
91
|
-
|
|
92
|
-
**Arguments**:
|
|
93
|
-
- `ANONYMIZED_FILE`: Path to the anonymized text file.
|
|
94
|
-
- `MAPPING_FILE`: Path to the JSON mapping file.
|
|
95
|
-
|
|
96
|
-
**Example**:
|
|
97
|
-
```bash
|
|
98
|
-
pdf-anonymizer deanonymize \
|
|
99
|
-
data/anonymized/document.anonymized.md \
|
|
100
|
-
data/mappings/document.mapping.json
|
|
101
|
-
```
|
|
102
|
-
|
|
103
|
-
This will create a deanonymized version of the file at `data/deanonymized/document.deanonymized.md`.
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
{pdf_anonymizer_cli-0.3.1 → pdf_anonymizer_cli-0.3.3}/src/pdf_anonymizer_cli.egg-info/SOURCES.txt
RENAMED
|
File without changes
|
|
File without changes
|
|
File without changes
|
{pdf_anonymizer_cli-0.3.1 → pdf_anonymizer_cli-0.3.3}/src/pdf_anonymizer_cli.egg-info/requires.txt
RENAMED
|
File without changes
|
{pdf_anonymizer_cli-0.3.1 → pdf_anonymizer_cli-0.3.3}/src/pdf_anonymizer_cli.egg-info/top_level.txt
RENAMED
|
File without changes
|