idscrub 1.1.2__tar.gz → 2.0.0__tar.gz

{idscrub-1.1.2 → idscrub-2.0.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: idscrub
-Version: 1.1.2
+Version: 2.0.0
 Author: Department for Business and Trade
 Classifier: Development Status :: 3 - Alpha
 Requires-Python: >=3.12
@@ -12,7 +12,8 @@ Requires-Dist: numpy>=2.3.4
 Requires-Dist: pandas<3.0
 Requires-Dist: phonenumbers>=9.0.18
 Requires-Dist: pip>=25.3
-Requires-Dist: spacy-transformers>=1.3.9
+Requires-Dist: spacy
+Requires-Dist: transformers
 Requires-Dist: tqdm>=4.67.1
 Requires-Dist: presidio-analyzer
 Requires-Dist: presidio-anonymizer
@@ -33,16 +34,20 @@ Dynamic: license-file
 
 ## Installation
 
-`idscrub` can be installed using `pip` into a Python **>=3.12** environment. Example:
+`idscrub` can be installed using `pip` into a Python **>=3.12** environment. 
+
+We recommend installing with the SpaCy transformer model (`en_core_web_trf`) as a dependency:
 
 ```console
-pip install idscrub
+pip install idscrub[trf]
 ```
-or with the spaCy transformer model (`en_core_web_trf`) already installed:
+
+If you do not need SpaCy:
 
 ```console
-pip install idscrub[trf]
+pip install idscrub
 ```
+
 ## How to use the code
 
 Basic usage example (see [basic_usage.ipynb](https://github.com/uktrade/idscrub/blob/main/notebooks/basic_usage.ipynb) for further examples):
@@ -50,18 +55,56 @@ Basic usage example (see [basic_usage.ipynb](https://github.com/uktrade/idscrub/
 ```python
 from idscrub import IDScrub
 
-scrub = IDScrub(['Our names are Hamish McDonald, L. Salah, and Elena Suárez.', 'My number is +441111111111 and I live at AA11 1AA.'])x
-scrubbed_texts = scrub.scrub(scrub_methods=['spacy_entities', 'uk_phone_numbers', 'uk_postcodes'])
+scrub = IDScrub(['Our names are Hamish McDonald, L. Salah, and Elena Suárez.', 'My number is +441111111111 and I live at AA11 1AA.'])
+scrubbed_texts = scrub.scrub(
+    pipeline=[
+        {"method": "spacy_entities", "entity_types": ["PERSON"]},
+        {"method": "uk_phone_numbers"},
+        {"method": "uk_postcodes"},
+    ]
+)
 
 print(scrubbed_texts)
 
 # Output: ['Our names are [PERSON], [PERSON], and [PERSON].', 'My number is [PHONENO] and I live at [POSTCODE].']
 ```
-## Personal data types supported
 
-Personal data can either be scrubbed as methods with arguments for extra customisation, e.g. `IDScrub.google_phone_numbers(region="GB")`, or as a string arguments with default configurations (see above). The method name and its string representation are the same. 
+This package will identify and scrub many types of data that you might not want to scrub, such as locations or context-relevent names. **We therefore highly recommend manually removing scrubbed data identified by `idscrub` from your original dataset on a case-by-case basis.**
+
+Scrubbed data can be identified using the following methods (see the [usage example notebook](https://github.com/uktrade/idscrub/blob/main/notebooks/basic_usage.ipynb) for further information):
 
-| Argument                | Scrubs                                                                 |
+```python
+import pandas as pd
+from idscrub import IDScrub
+
+# From lists of text:
+scrub = IDScrub(['Our names are Hamish McDonald, L. Salah, and Elena Suárez.', 'My number is +441111111111 and I live at AA11 1AA.'])
+scrubbed_texts = scrub.scrub(
+    pipeline=[
+        {"method": "spacy_entities", "entity_types": ["PERSON"]},
+        {"method": "uk_phone_numbers"},
+        {"method": "uk_postcodes"},
+    ]
+)
+scrubbed_df = scrub.get_scrubbed_data()
+print(scrubbed_df)
+
+# From a Pandas DataFrame:
+scrubbed_df, scrubbed_data = IDScrub.dataframe(
+    df=pd.read_csv('path/to/csv'), 
+    id_col="ID", 
+    pipeline=[
+        {"method": "spacy_entities", "entity_types": ["PERSON"]},
+        {"method": "uk_phone_numbers"},
+        {"method": "uk_postcodes"},
+    ]
+)
+print(scrubbed_df)
+```
+
+## Personal data types supported
+
+| Method                | Scrubs                                                                 |
 |-------------------------|------------------------------------------------------------------------|
 | `all`                  | All supported personal data types (see `IDScrub.all()` for further customisation) |
 | `spacy_entities`        | Entities detected by spaCy's `en_core_web_trf` or other user-selected spaCy models (e.g. persons (names), organisations) |
@@ -70,12 +113,15 @@ Personal data can either be scrubbed as methods with arguments for extra customi
 | `email_addresses`      | Email addresses (e.g. john@email.com)   |
 | `titles`               | Titles (e.g. Mr., Mrs., Dr.)    |
 | `handles`              | Social media handles (e.g. @username)  |
+| `urls`                 | URLs (e.g. www.bbc.co.uk) |
 | `ip_addresses`         | IP addresses (e.g. 8.8.8.8)  |
 | `uk_postcodes`         | UK postal codes (e.g. SW1A 2AA) |
 | `uk_addresses`         | UK addresses (e.g. 10 Downing Street)  |
 | `uk_phone_numbers`     | UK phone numbers (e.g. +441111111111) |
 | `google_phone_numbers` | Phone numbers detected by Google's [phonenumbers](https://github.com/daviddrysdale/python-phonenumbers) |
 
+Method arguments for further customisation can be viewed by viewing the docstring e.g. `?IDScrub.spacy_entities`.
+
 ## Considerations before use
 
 - You must follow [GDPR guidance](https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/the-research-provisions/principles-and-grounds-for-processing/) when processing personal data using this package.
@@ -130,7 +176,7 @@ This project is managed by [uv](https://docs.astral.sh/uv/).
 To install all dependencies for this project, run:
 
 ```console
-uv sync --all-extras
+uv sync
 ```
 
 If you do not have Python 3.12, run:

{idscrub-1.1.2 → idscrub-2.0.0}/README.md

@@ -11,16 +11,20 @@
 
 ## Installation
 
-`idscrub` can be installed using `pip` into a Python **>=3.12** environment. Example:
+`idscrub` can be installed using `pip` into a Python **>=3.12** environment. 
+
+We recommend installing with the SpaCy transformer model (`en_core_web_trf`) as a dependency:
 
 ```console
-pip install idscrub
+pip install idscrub[trf]
 ```
-or with the spaCy transformer model (`en_core_web_trf`) already installed:
+
+If you do not need SpaCy:
 
 ```console
-pip install idscrub[trf]
+pip install idscrub
 ```
+
 ## How to use the code
 
 Basic usage example (see [basic_usage.ipynb](https://github.com/uktrade/idscrub/blob/main/notebooks/basic_usage.ipynb) for further examples):
@@ -28,18 +32,56 @@ Basic usage example (see [basic_usage.ipynb](https://github.com/uktrade/idscrub/
 ```python
 from idscrub import IDScrub
 
-scrub = IDScrub(['Our names are Hamish McDonald, L. Salah, and Elena Suárez.', 'My number is +441111111111 and I live at AA11 1AA.'])x
-scrubbed_texts = scrub.scrub(scrub_methods=['spacy_entities', 'uk_phone_numbers', 'uk_postcodes'])
+scrub = IDScrub(['Our names are Hamish McDonald, L. Salah, and Elena Suárez.', 'My number is +441111111111 and I live at AA11 1AA.'])
+scrubbed_texts = scrub.scrub(
+    pipeline=[
+        {"method": "spacy_entities", "entity_types": ["PERSON"]},
+        {"method": "uk_phone_numbers"},
+        {"method": "uk_postcodes"},
+    ]
+)
 
 print(scrubbed_texts)
 
 # Output: ['Our names are [PERSON], [PERSON], and [PERSON].', 'My number is [PHONENO] and I live at [POSTCODE].']
 ```
-## Personal data types supported
 
-Personal data can either be scrubbed as methods with arguments for extra customisation, e.g. `IDScrub.google_phone_numbers(region="GB")`, or as a string arguments with default configurations (see above). The method name and its string representation are the same. 
+This package will identify and scrub many types of data that you might not want to scrub, such as locations or context-relevent names. **We therefore highly recommend manually removing scrubbed data identified by `idscrub` from your original dataset on a case-by-case basis.**
+
+Scrubbed data can be identified using the following methods (see the [usage example notebook](https://github.com/uktrade/idscrub/blob/main/notebooks/basic_usage.ipynb) for further information):
 
-| Argument                | Scrubs                                                                 |
+```python
+import pandas as pd
+from idscrub import IDScrub
+
+# From lists of text:
+scrub = IDScrub(['Our names are Hamish McDonald, L. Salah, and Elena Suárez.', 'My number is +441111111111 and I live at AA11 1AA.'])
+scrubbed_texts = scrub.scrub(
+    pipeline=[
+        {"method": "spacy_entities", "entity_types": ["PERSON"]},
+        {"method": "uk_phone_numbers"},
+        {"method": "uk_postcodes"},
+    ]
+)
+scrubbed_df = scrub.get_scrubbed_data()
+print(scrubbed_df)
+
+# From a Pandas DataFrame:
+scrubbed_df, scrubbed_data = IDScrub.dataframe(
+    df=pd.read_csv('path/to/csv'), 
+    id_col="ID", 
+    pipeline=[
+        {"method": "spacy_entities", "entity_types": ["PERSON"]},
+        {"method": "uk_phone_numbers"},
+        {"method": "uk_postcodes"},
+    ]
+)
+print(scrubbed_df)
+```
+
+## Personal data types supported
+
+| Method                | Scrubs                                                                 |
 |-------------------------|------------------------------------------------------------------------|
 | `all`                  | All supported personal data types (see `IDScrub.all()` for further customisation) |
 | `spacy_entities`        | Entities detected by spaCy's `en_core_web_trf` or other user-selected spaCy models (e.g. persons (names), organisations) |
@@ -48,12 +90,15 @@ Personal data can either be scrubbed as methods with arguments for extra customi
 | `email_addresses`      | Email addresses (e.g. john@email.com)   |
 | `titles`               | Titles (e.g. Mr., Mrs., Dr.)    |
 | `handles`              | Social media handles (e.g. @username)  |
+| `urls`                 | URLs (e.g. www.bbc.co.uk) |
 | `ip_addresses`         | IP addresses (e.g. 8.8.8.8)  |
 | `uk_postcodes`         | UK postal codes (e.g. SW1A 2AA) |
 | `uk_addresses`         | UK addresses (e.g. 10 Downing Street)  |
 | `uk_phone_numbers`     | UK phone numbers (e.g. +441111111111) |
 | `google_phone_numbers` | Phone numbers detected by Google's [phonenumbers](https://github.com/daviddrysdale/python-phonenumbers) |
 
+Method arguments for further customisation can be viewed by viewing the docstring e.g. `?IDScrub.spacy_entities`.
+
 ## Considerations before use
 
 - You must follow [GDPR guidance](https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/the-research-provisions/principles-and-grounds-for-processing/) when processing personal data using this package.
@@ -108,7 +153,7 @@ This project is managed by [uv](https://docs.astral.sh/uv/).
 To install all dependencies for this project, run:
 
 ```console
-uv sync --all-extras
+uv sync
 ```
 
 If you do not have Python 3.12, run: