idscrub 1.1.2__py3-none-any.whl → 2.0.0__py3-none-any.whl

{idscrub-1.1.2.dist-info → idscrub-2.0.0.dist-info}/METADATA

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: idscrub
-Version: 1.1.2
+Version: 2.0.0
 Author: Department for Business and Trade
 Classifier: Development Status :: 3 - Alpha
 Requires-Python: >=3.12
@@ -12,7 +12,8 @@ Requires-Dist: numpy>=2.3.4
 Requires-Dist: pandas<3.0
 Requires-Dist: phonenumbers>=9.0.18
 Requires-Dist: pip>=25.3
-Requires-Dist: spacy-transformers>=1.3.9
+Requires-Dist: spacy
+Requires-Dist: transformers
 Requires-Dist: tqdm>=4.67.1
 Requires-Dist: presidio-analyzer
 Requires-Dist: presidio-anonymizer
@@ -33,16 +34,20 @@ Dynamic: license-file
 
 ## Installation
 
-`idscrub` can be installed using `pip` into a Python **>=3.12** environment. Example:
+`idscrub` can be installed using `pip` into a Python **>=3.12** environment. 
+
+We recommend installing with the SpaCy transformer model (`en_core_web_trf`) as a dependency:
 
 ```console
-pip install idscrub
+pip install idscrub[trf]
 ```
-or with the spaCy transformer model (`en_core_web_trf`) already installed:
+
+If you do not need SpaCy:
 
 ```console
-pip install idscrub[trf]
+pip install idscrub
 ```
+
 ## How to use the code
 
 Basic usage example (see [basic_usage.ipynb](https://github.com/uktrade/idscrub/blob/main/notebooks/basic_usage.ipynb) for further examples):
@@ -50,18 +55,56 @@ Basic usage example (see [basic_usage.ipynb](https://github.com/uktrade/idscrub/
 ```python
 from idscrub import IDScrub
 
-scrub = IDScrub(['Our names are Hamish McDonald, L. Salah, and Elena Suárez.', 'My number is +441111111111 and I live at AA11 1AA.'])x
-scrubbed_texts = scrub.scrub(scrub_methods=['spacy_entities', 'uk_phone_numbers', 'uk_postcodes'])
+scrub = IDScrub(['Our names are Hamish McDonald, L. Salah, and Elena Suárez.', 'My number is +441111111111 and I live at AA11 1AA.'])
+scrubbed_texts = scrub.scrub(
+    pipeline=[
+        {"method": "spacy_entities", "entity_types": ["PERSON"]},
+        {"method": "uk_phone_numbers"},
+        {"method": "uk_postcodes"},
+    ]
+)
 
 print(scrubbed_texts)
 
 # Output: ['Our names are [PERSON], [PERSON], and [PERSON].', 'My number is [PHONENO] and I live at [POSTCODE].']
 ```
-## Personal data types supported
 
-Personal data can either be scrubbed as methods with arguments for extra customisation, e.g. `IDScrub.google_phone_numbers(region="GB")`, or as a string arguments with default configurations (see above). The method name and its string representation are the same. 
+This package will identify and scrub many types of data that you might not want to scrub, such as locations or context-relevent names. **We therefore highly recommend manually removing scrubbed data identified by `idscrub` from your original dataset on a case-by-case basis.**
+
+Scrubbed data can be identified using the following methods (see the [usage example notebook](https://github.com/uktrade/idscrub/blob/main/notebooks/basic_usage.ipynb) for further information):
 
-| Argument                | Scrubs                                                                 |
+```python
+import pandas as pd
+from idscrub import IDScrub
+
+# From lists of text:
+scrub = IDScrub(['Our names are Hamish McDonald, L. Salah, and Elena Suárez.', 'My number is +441111111111 and I live at AA11 1AA.'])
+scrubbed_texts = scrub.scrub(
+    pipeline=[
+        {"method": "spacy_entities", "entity_types": ["PERSON"]},
+        {"method": "uk_phone_numbers"},
+        {"method": "uk_postcodes"},
+    ]
+)
+scrubbed_df = scrub.get_scrubbed_data()
+print(scrubbed_df)
+
+# From a Pandas DataFrame:
+scrubbed_df, scrubbed_data = IDScrub.dataframe(
+    df=pd.read_csv('path/to/csv'), 
+    id_col="ID", 
+    pipeline=[
+        {"method": "spacy_entities", "entity_types": ["PERSON"]},
+        {"method": "uk_phone_numbers"},
+        {"method": "uk_postcodes"},
+    ]
+)
+print(scrubbed_df)
+```
+
+## Personal data types supported
+
+| Method                | Scrubs                                                                 |
 |-------------------------|------------------------------------------------------------------------|
 | `all`                  | All supported personal data types (see `IDScrub.all()` for further customisation) |
 | `spacy_entities`        | Entities detected by spaCy's `en_core_web_trf` or other user-selected spaCy models (e.g. persons (names), organisations) |
@@ -70,12 +113,15 @@ Personal data can either be scrubbed as methods with arguments for extra customi
 | `email_addresses`      | Email addresses (e.g. john@email.com)   |
 | `titles`               | Titles (e.g. Mr., Mrs., Dr.)    |
 | `handles`              | Social media handles (e.g. @username)  |
+| `urls`                 | URLs (e.g. www.bbc.co.uk) |
 | `ip_addresses`         | IP addresses (e.g. 8.8.8.8)  |
 | `uk_postcodes`         | UK postal codes (e.g. SW1A 2AA) |
 | `uk_addresses`         | UK addresses (e.g. 10 Downing Street)  |
 | `uk_phone_numbers`     | UK phone numbers (e.g. +441111111111) |
 | `google_phone_numbers` | Phone numbers detected by Google's [phonenumbers](https://github.com/daviddrysdale/python-phonenumbers) |
 
+Method arguments for further customisation can be viewed by viewing the docstring e.g. `?IDScrub.spacy_entities`.
+
 ## Considerations before use
 
 - You must follow [GDPR guidance](https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/the-research-provisions/principles-and-grounds-for-processing/) when processing personal data using this package.
@@ -130,7 +176,7 @@ This project is managed by [uv](https://docs.astral.sh/uv/).
 To install all dependencies for this project, run:
 
 ```console
-uv sync --all-extras
+uv sync
 ```
 
 If you do not have Python 3.12, run:

idscrub-2.0.0.dist-info/RECORD

@@ -0,0 +1,24 @@
+idscrub/__init__.py,sha256=cRugJv27q1q--bl-VNLpfiScJb_ROlUxyLFhaF55S1w,38
+idscrub/locations.py,sha256=7fMNOcGMYe7sX8TrfhMW6oYGAlc1WVYVQKQbpxE3pqo,217
+idscrub/scrub.py,sha256=ow5gDejdchXxi3iN1vEqlCaJGGkaEnvFJ8nAau6w9XI,45308
+idscrub-2.0.0.dist-info/licenses/LICENSE,sha256=JJnuf10NSx7YXglte1oH_N9ZP3AcWR_Y8irvQb_wnsg,1090
+notebooks/basic_usage.ipynb,sha256=kfCgdN8DnjIUd9wZs93mpLKMT5NA3E8ty8a6mQUAZGo,40195
+test/conftest.py,sha256=CD_fYlo-qjkDgsW-ZRMZjW5Famqpt8fzN2HOcJ2MBDU,1450
+test/test_dataframe.py,sha256=MWHQVJ_lCKLR86lnGGz9Nlke-pPQwvim11Cs5INxmh8,6395
+test/test_errors.py,sha256=IA4vuYUfq_sEhHK-6Ba25y8WjUfHzODio53zjhLBKCU,944
+test/test_exclude.py,sha256=TuO4W6TUVZyPurPst1ZSbqHCRFCvrruLkNSoxks_arI,601
+test/test_group.py,sha256=um6UlWeUx94cDADoNd525z8-Uw02oO4clB2hP6iy4PY,235
+test/test_huggingface.py,sha256=awIhTN4iBErvm1Mt6ZAHlgCBqSeSCl12X6t7sMtprD0,935
+test/test_id.py,sha256=z2Za6sk-Km-r2FN6i03_q2IY5I21OkwfOtc4p9sRvJg,1108
+test/test_label.py,sha256=uiUF72nZFt5GWE7HWO2ura3XU4hSKfe6ZTOyc7oRyzg,1053
+test/test_overlap.py,sha256=kM6jePx94evk-VRkMzeH_UiuYPR7FFe63oBgLsvfvZw,2380
+test/test_phonenumbers.py,sha256=LqAUzSYdYoQQyO_mAHYQkkRCgh4uArX4oxxKVhMi548,661
+test/test_presidio.py,sha256=Y6kgPOZ_ie3shVThbNfJ2i7w1yifS5ukI4eAo1Lgm6k,1658
+test/test_regex.py,sha256=Z_D8ttPYnRCr4gl13W9IEb7vS6QtWMReAc9Rufbtv50,6789
+test/test_scrub.py,sha256=aelzLpF2S_5F754VcZtWeEzQIcUZ0ex0m6A8mTAnWQg,1591
+test/test_scrub_text.py,sha256=WadP66U3jrSqXlbUrIrguDYSfhRKFJW73lR6eudC22o,605
+test/test_spacy.py,sha256=5wIPnHirs6fx1Msz9IDDF8dxyh3t1asWaFEvSXeiBvU,1910
+idscrub-2.0.0.dist-info/METADATA,sha256=ARod5-Azd--n-lWnZ3yHF4jt9MSjpPrcnMePzmpp8BM,8537
+idscrub-2.0.0.dist-info/WHEEL,sha256=wUyA8OaulRlbfwMtmQsvNngGrxQHAvkKcvRmdizlJi0,92
+idscrub-2.0.0.dist-info/top_level.txt,sha256=D4EEodXGCjGiX35ObiBTmjjBAdouN-eCvH-LezGGtks,23
+idscrub-2.0.0.dist-info/RECORD,,