idscrub 0.2.1__py3-none-any.whl → 0.2.2__py3-none-any.whl

{idscrub-0.2.1.dist-info → idscrub-0.2.2.dist-info}/METADATA

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: idscrub
-Version: 0.2.1
+Version: 0.2.2
 Author: Department for Business and Trade
 Requires-Python: >=3.12
 Description-Content-Type: text/markdown
@@ -21,36 +21,45 @@ Dynamic: license-file
 
 # idscrub 🧽✨
 
-## Project Information
+* Names and other personally identifying information are often present in text.
+* This information may need to be removed prior to further analysis in many cases.
+* `idscrub` identifies and removes (*✨scrubs✨*) personal data from text using [regular expressions](https://en.wikipedia.org/wiki/Regular_expression) and [named-entity recognition](https://en.wikipedia.org/wiki/Named-entity_recognition).
 
-* This package removes (*✨scrubs✨*) identifying personal data from text using [regular expressions](https://en.wikipedia.org/wiki/Regular_expression) and [named-entity recognition](https://en.wikipedia.org/wiki/Named-entity_recognition).
+## Installation
 
-> [!WARNING]
-> You must follow [GDPR guidance](https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/the-research-provisions/principles-and-grounds-for-processing/) when processing personal data using this package.
->
-> Specifically, you must:
->
-> - **Update privacy notices**: Clearly state this processing activity in new or existing privacy notices before using the package.
-> - **Ensure secure deletion**: Remove any temporary or intermediary files and outputs in a secure manner.
-> - **Ensure data subject rights upheld**: Ensure individuals can access, correct, or erase their data as required.
-> - **Maintain processing records**: Document how personal data is handled and for what purpose.
+`idscrub` can be installed using `pip` into a Python **>=3.12** environment. Example:
 
-### Description
+```console
+pip install idscrub
+```
+or with the spaCy transformer model (`en_core_web_trf`) already installed:
 
-* Names and other personally identifying information are often present in text.
-* This information may need to be removed prior to further analysis in many cases.
-* `idscrub` provides a standardised way to do this in the Department for Business and Trade. 
+```console
+pip install idscrub[trf]
+```
+## How to use the code
 
-### Expected Outputs
+Basic usage example (see [basic_usage.ipynb](https://github.com/uktrade/idscrub/blob/main/notebooks/basic_usage.ipynb) for further examples):
 
-* A list of text with names and other identifying information removed.
+```python
+from idscrub import IDScrub
 
-> [!WARNING]
-> * This package has been designed as a *first pass* for standardised personal data removal. 
-> * Users are encouraged to check and confirm outputs and conduct manual reviews where necessary, e.g. when cleaning high risk datasets.
-> * It is up to the user to assess whether this removal process needs to be supplemented by other methods for their given dataset and security requirements.
+scrub = IDScrub(['Our names are Hamish McDonald, L. Salah, and Elena Suárez.', 'My number is +441111111111 and I live at AA11 1AA.'])x
+scrubbed_texts = scrub.scrub(scrub_methods=['spacy_persons', 'uk_phone_numbers', 'uk_postcodes'])
 
-### Data
+print(scrubbed_texts)
+
+# Output: ['Our names are [PERSON], [PERSON], and [PERSON].', 'My number is [PHONENO] and I live at [POSTCODE].']
+```
+
+## Considerations before use
+
+- You must follow [GDPR guidance](https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/the-research-provisions/principles-and-grounds-for-processing/) when processing personal data using this package.
+- This package has been designed as a *first pass* for standardised personal data removal. 
+- Users are encouraged to check and confirm outputs and conduct manual reviews where necessary, e.g. when cleaning high risk datasets.
+- It is up to the user to assess whether this removal process needs to be supplemented by other methods for their given dataset and security requirements.
+
+### Input data
 
 - This package is designed for text-based documents structured as a list of strings.
 - It performs best when contextual meaning can be inferred from the text.
@@ -68,50 +77,25 @@ Dynamic: license-file
 > [!IMPORTANT]
 > * See [our wiki](https://github.com/uktrade/idscrub/wiki/Evaluation) for further details and notes on our evaluation of `idscrub`.
 
-### Models and Memory
+### Models
 
 * Only Spacy's `en_core_web_trf` and no Hugging Face models have been formally evaluated.
 * We therefore recommend that the current default `en_core_web_trf` is used for name scrubbing. **Other models need to be evaluated by the user.**
 
-> [!IMPORTANT]
-> Spacy and Hugging Face models have high memory requirements. To avoid memory-related errors. Clear the auto-generated `huggingface` folder if not in use. Do not push the `huggingface` folder (or user-defined equivalent) to GitHub.
-
 ## Similar Python packages
 
-* Similar packages exist for undertaking this task, such as [presidio](https://microsoft.github.io/presidio/), [scrubadub](https://github.com/LeapBeyond/scrubadub) and [sanityze](https://github.com/UBC-MDS/sanityze). 
-* Development of `idscrub` was undertaken to: bring together different scrubbing methods across the department, adhere to infrastructure requirements, guarantee future stability and maintainability, and encourage future scrubbing methods to be added collaboratively and transparently. 
-* To leverage the power of other packages, we have added methods that allow you to interact with them. These include: `IDScrub.presidio()` and `IDScrub.google_phone_numbers()`. See the [usage example notebook](https://github.com/uktrade/idscrub/blob/main/notebooks/basic_usage.ipynb) and method docstrings for further information.
-
-
-## Installation
-
-`idscrub` can be installed using `pip` into a Python **>=3.12** environment. Example:
-
-```console
-pip install idscrub
-```
-or with the spaCy transformer model (`en_core_web_trf`) already installed:
-
-```console
-pip instll idscrub[trf]
-```
-
-## How to use the code
-
-Basic usage example (see `notebooks/basic_usage.ipynb` for further examples):
+* Similar packages exist for undertaking this task, such as [Presidio](https://microsoft.github.io/presidio/), [Scrubadub](https://github.com/LeapBeyond/scrubadub) and [Sanityze](https://github.com/UBC-MDS/sanityze). 
+* Development of `idscrub` was undertaken to: 
 
-```python
-from idscrub import IDScrub
-
-scrub = IDScrub(['Our names are Hamish McDonald, L. Salah, and Elena Suárez.', 'My number is +441111111111 and I live at AA11 1AA.'])
-scrubbed_texts = scrub.scrub(scrub_methods=['spacy_persons', 'uk_phone_numbers', 'uk_postcodes'])
-
-print(scrubbed_texts)
-
-# Output: ['Our names are [PERSON], [PERSON], and [PERSON].', 'My number is [PHONENO] and I live at [POSTCODE].']
-```
+    * Bring together different scrubbing methods across the Department for Business and Trade.
+    * Adhere to infrastructure requirements.
+    * Guarantee future stability and maintainability.
+    * Encourage future scrubbing methods to be added collaboratively and transparently.
+    * Allow for full flexibility depending on the use case and required outputs.
+    
+* To leverage the power of other packages, we have added methods that allow you to interact with them. These include: `IDScrub.presidio()` and `IDScrub.google_phone_numbers()`. See the [usage example notebook](https://github.com/uktrade/idscrub/blob/main/notebooks/basic_usage.ipynb) and method docstrings for further information.
 
-## AI Declaration
+## AI declaration
 
 AI has been used in the development of `idscrub`, primarily to develop regular expressions, suggest code refinements and draft documentation.
 

{idscrub-0.2.1.dist-info → idscrub-0.2.2.dist-info}/RECORD

@@ -1,7 +1,7 @@
 idscrub/__init__.py,sha256=cRugJv27q1q--bl-VNLpfiScJb_ROlUxyLFhaF55S1w,38
 idscrub/locations.py,sha256=7fMNOcGMYe7sX8TrfhMW6oYGAlc1WVYVQKQbpxE3pqo,217
 idscrub/scrub.py,sha256=K4Sw4DxKhYJnnu_vpRhUcqj-AbeGr8SwDB0XrDLEciM,34940
-idscrub-0.2.1.dist-info/licenses/LICENSE,sha256=JJnuf10NSx7YXglte1oH_N9ZP3AcWR_Y8irvQb_wnsg,1090
+idscrub-0.2.2.dist-info/licenses/LICENSE,sha256=JJnuf10NSx7YXglte1oH_N9ZP3AcWR_Y8irvQb_wnsg,1090
 notebooks/basic_usage.ipynb,sha256=eQFU3mOyRXbCwFz3jVUKCxWRtIP5Jptny8fj-KYoBwA,39784
 test/conftest.py,sha256=ph1S3LMvzlzvOsb3l2YhpyHSdmg4uV7p61ge_JVCGv0,267
 test/test_all.py,sha256=z6v9O2Ts9dWITlhvZwRMyKUZsO7ncaT3znqqBCKJ6Wc,1141
@@ -15,7 +15,7 @@ test/test_phonenumbers.py,sha256=hZsXgwhn5R-7426TTWwCH9gWQwhyHtjLUstN10jnX6c,607
 test/test_regex.py,sha256=EQGx3PHwJJzIdy6xwR8gEsSRDtlWHR-U81EPI811eZA,4474
 test/test_scrub.py,sha256=pohmw3frtlkmZDMvOEbmvVJgtcVdFlEDL3TxR5-y-0Q,1422
 test/test_spacy.py,sha256=mrUGUulvzDGgQRttdG0tgL2sGBRmYfg1fDNp7SFq8as,961
-idscrub-0.2.1.dist-info/METADATA,sha256=7F8667RcPvLjVeCBxEeisoLYsy8r5H-xqwatScbxknk,6127
-idscrub-0.2.1.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
-idscrub-0.2.1.dist-info/top_level.txt,sha256=D4EEodXGCjGiX35ObiBTmjjBAdouN-eCvH-LezGGtks,23
-idscrub-0.2.1.dist-info/RECORD,,
+idscrub-0.2.2.dist-info/METADATA,sha256=IHoFTVY6cJARkeeKoQlpunA7Nboc4y32bpSoS-IgSoM,5352
+idscrub-0.2.2.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
+idscrub-0.2.2.dist-info/top_level.txt,sha256=D4EEodXGCjGiX35ObiBTmjjBAdouN-eCvH-LezGGtks,23
+idscrub-0.2.2.dist-info/RECORD,,