PyPI - idscrub - Versions diffs - 0.2.1__tar.gz → 0.2.2__tar.gz - Mend

idscrub 0.2.1tar.gz → 0.2.2tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (36) hide show

{idscrub-0.2.1 → idscrub-0.2.2}/PKG-INFO RENAMED Viewed

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: idscrub
-Version: 0.2.1
+Version: 0.2.2
 Author: Department for Business and Trade
 Requires-Python: >=3.12
 Description-Content-Type: text/markdown
@@ -21,36 +21,45 @@ Dynamic: license-file
 # idscrub 🧽✨
-## Project Information
+* Names and other personally identifying information are often present in text.
+* This information may need to be removed prior to further analysis in many cases.
+* `idscrub` identifies and removes (*✨scrubs✨*) personal data from text using [regular expressions](https://en.wikipedia.org/wiki/Regular_expression) and [named-entity recognition](https://en.wikipedia.org/wiki/Named-entity_recognition).
-* This package removes (*✨scrubs✨*) identifying personal data from text using [regular expressions](https://en.wikipedia.org/wiki/Regular_expression) and [named-entity recognition](https://en.wikipedia.org/wiki/Named-entity_recognition).
+## Installation
-> [!WARNING]
-> You must follow [GDPR guidance](https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/the-research-provisions/principles-and-grounds-for-processing/) when processing personal data using this package.
->
-> Specifically, you must:
->
-> - **Update privacy notices**: Clearly state this processing activity in new or existing privacy notices before using the package.
-> - **Ensure secure deletion**: Remove any temporary or intermediary files and outputs in a secure manner.
-> - **Ensure data subject rights upheld**: Ensure individuals can access, correct, or erase their data as required.
-> - **Maintain processing records**: Document how personal data is handled and for what purpose.
+`idscrub` can be installed using `pip` into a Python **>=3.12** environment. Example:
-### Description
+```console
+pip install idscrub
+```
+or with the spaCy transformer model (`en_core_web_trf`) already installed:
-* Names and other personally identifying information are often present in text.
-* This information may need to be removed prior to further analysis in many cases.
-* `idscrub` provides a standardised way to do this in the Department for Business and Trade.
+```console
+pip install idscrub[trf]
+```
+## How to use the code
-### Expected Outputs
+Basic usage example (see [basic_usage.ipynb](https://github.com/uktrade/idscrub/blob/main/notebooks/basic_usage.ipynb) for further examples):
-* A list of text with names and other identifying information removed.
+```python
+from idscrub import IDScrub
-> [!WARNING]
-> * This package has been designed as a *first pass* for standardised personal data removal.
-> * Users are encouraged to check and confirm outputs and conduct manual reviews where necessary, e.g. when cleaning high risk datasets.
-> * It is up to the user to assess whether this removal process needs to be supplemented by other methods for their given dataset and security requirements.
+scrub = IDScrub(['Our names are Hamish McDonald, L. Salah, and Elena Suárez.', 'My number is +441111111111 and I live at AA11 1AA.'])x
+scrubbed_texts = scrub.scrub(scrub_methods=['spacy_persons', 'uk_phone_numbers', 'uk_postcodes'])
-### Data
+print(scrubbed_texts)
+# Output: ['Our names are [PERSON], [PERSON], and [PERSON].', 'My number is [PHONENO] and I live at [POSTCODE].']
+```
+## Considerations before use
+- You must follow [GDPR guidance](https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/the-research-provisions/principles-and-grounds-for-processing/) when processing personal data using this package.
+- This package has been designed as a *first pass* for standardised personal data removal.
+- Users are encouraged to check and confirm outputs and conduct manual reviews where necessary, e.g. when cleaning high risk datasets.
+- It is up to the user to assess whether this removal process needs to be supplemented by other methods for their given dataset and security requirements.
+### Input data
 - This package is designed for text-based documents structured as a list of strings.
 - It performs best when contextual meaning can be inferred from the text.
@@ -68,50 +77,25 @@ Dynamic: license-file
 > [!IMPORTANT]
 > * See [our wiki](https://github.com/uktrade/idscrub/wiki/Evaluation) for further details and notes on our evaluation of `idscrub`.
-### Models and Memory
+### Models
 * Only Spacy's `en_core_web_trf` and no Hugging Face models have been formally evaluated.
 * We therefore recommend that the current default `en_core_web_trf` is used for name scrubbing. **Other models need to be evaluated by the user.**
-> [!IMPORTANT]
-> Spacy and Hugging Face models have high memory requirements. To avoid memory-related errors. Clear the auto-generated `huggingface` folder if not in use. Do not push the `huggingface` folder (or user-defined equivalent) to GitHub.
 ## Similar Python packages
-* Similar packages exist for undertaking this task, such as [presidio](https://microsoft.github.io/presidio/), [scrubadub](https://github.com/LeapBeyond/scrubadub) and [sanityze](https://github.com/UBC-MDS/sanityze).
-* Development of `idscrub` was undertaken to: bring together different scrubbing methods across the department, adhere to infrastructure requirements, guarantee future stability and maintainability, and encourage future scrubbing methods to be added collaboratively and transparently.
-* To leverage the power of other packages, we have added methods that allow you to interact with them. These include: `IDScrub.presidio()` and `IDScrub.google_phone_numbers()`. See the [usage example notebook](https://github.com/uktrade/idscrub/blob/main/notebooks/basic_usage.ipynb) and method docstrings for further information.
-## Installation
-`idscrub` can be installed using `pip` into a Python **>=3.12** environment. Example:
-```console
-pip install idscrub
-```
-or with the spaCy transformer model (`en_core_web_trf`) already installed:
-```console
-pip instll idscrub[trf]
-```
-## How to use the code
-Basic usage example (see `notebooks/basic_usage.ipynb` for further examples):
+* Similar packages exist for undertaking this task, such as [Presidio](https://microsoft.github.io/presidio/), [Scrubadub](https://github.com/LeapBeyond/scrubadub) and [Sanityze](https://github.com/UBC-MDS/sanityze).
+* Development of `idscrub` was undertaken to:
-```python
-from idscrub import IDScrub
-scrub = IDScrub(['Our names are Hamish McDonald, L. Salah, and Elena Suárez.', 'My number is +441111111111 and I live at AA11 1AA.'])
-scrubbed_texts = scrub.scrub(scrub_methods=['spacy_persons', 'uk_phone_numbers', 'uk_postcodes'])
-print(scrubbed_texts)
-# Output: ['Our names are [PERSON], [PERSON], and [PERSON].', 'My number is [PHONENO] and I live at [POSTCODE].']
-```
+    * Bring together different scrubbing methods across the Department for Business and Trade.
+    * Adhere to infrastructure requirements.
+    * Guarantee future stability and maintainability.
+    * Encourage future scrubbing methods to be added collaboratively and transparently.
+    * Allow for full flexibility depending on the use case and required outputs.
+* To leverage the power of other packages, we have added methods that allow you to interact with them. These include: `IDScrub.presidio()` and `IDScrub.google_phone_numbers()`. See the [usage example notebook](https://github.com/uktrade/idscrub/blob/main/notebooks/basic_usage.ipynb) and method docstrings for further information.
-## AI Declaration
+## AI declaration
 AI has been used in the development of `idscrub`, primarily to develop regular expressions, suggest code refinements and draft documentation.

{idscrub-0.2.1 → idscrub-0.2.2}/README.md RENAMED Viewed

@@ -1,35 +1,44 @@
 # idscrub 🧽✨
-## Project Information
+* Names and other personally identifying information are often present in text.
+* This information may need to be removed prior to further analysis in many cases.
+* `idscrub` identifies and removes (*✨scrubs✨*) personal data from text using [regular expressions](https://en.wikipedia.org/wiki/Regular_expression) and [named-entity recognition](https://en.wikipedia.org/wiki/Named-entity_recognition).
-* This package removes (*✨scrubs✨*) identifying personal data from text using [regular expressions](https://en.wikipedia.org/wiki/Regular_expression) and [named-entity recognition](https://en.wikipedia.org/wiki/Named-entity_recognition).
+## Installation
-> [!WARNING]
-> You must follow [GDPR guidance](https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/the-research-provisions/principles-and-grounds-for-processing/) when processing personal data using this package.
->
-> Specifically, you must:
->
-> - **Update privacy notices**: Clearly state this processing activity in new or existing privacy notices before using the package.
-> - **Ensure secure deletion**: Remove any temporary or intermediary files and outputs in a secure manner.
-> - **Ensure data subject rights upheld**: Ensure individuals can access, correct, or erase their data as required.
-> - **Maintain processing records**: Document how personal data is handled and for what purpose.
+`idscrub` can be installed using `pip` into a Python **>=3.12** environment. Example:
-### Description
+```console
+pip install idscrub
+```
+or with the spaCy transformer model (`en_core_web_trf`) already installed:
-* Names and other personally identifying information are often present in text.
-* This information may need to be removed prior to further analysis in many cases.
-* `idscrub` provides a standardised way to do this in the Department for Business and Trade.
+```console
+pip install idscrub[trf]
+```
+## How to use the code
-### Expected Outputs
+Basic usage example (see [basic_usage.ipynb](https://github.com/uktrade/idscrub/blob/main/notebooks/basic_usage.ipynb) for further examples):
-* A list of text with names and other identifying information removed.
+```python
+from idscrub import IDScrub
-> [!WARNING]
-> * This package has been designed as a *first pass* for standardised personal data removal.
-> * Users are encouraged to check and confirm outputs and conduct manual reviews where necessary, e.g. when cleaning high risk datasets.
-> * It is up to the user to assess whether this removal process needs to be supplemented by other methods for their given dataset and security requirements.
+scrub = IDScrub(['Our names are Hamish McDonald, L. Salah, and Elena Suárez.', 'My number is +441111111111 and I live at AA11 1AA.'])x
+scrubbed_texts = scrub.scrub(scrub_methods=['spacy_persons', 'uk_phone_numbers', 'uk_postcodes'])
-### Data
+print(scrubbed_texts)
+# Output: ['Our names are [PERSON], [PERSON], and [PERSON].', 'My number is [PHONENO] and I live at [POSTCODE].']
+```
+## Considerations before use
+- You must follow [GDPR guidance](https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/the-research-provisions/principles-and-grounds-for-processing/) when processing personal data using this package.
+- This package has been designed as a *first pass* for standardised personal data removal.
+- Users are encouraged to check and confirm outputs and conduct manual reviews where necessary, e.g. when cleaning high risk datasets.
+- It is up to the user to assess whether this removal process needs to be supplemented by other methods for their given dataset and security requirements.
+### Input data
 - This package is designed for text-based documents structured as a list of strings.
 - It performs best when contextual meaning can be inferred from the text.
@@ -47,50 +56,25 @@
 > [!IMPORTANT]
 > * See [our wiki](https://github.com/uktrade/idscrub/wiki/Evaluation) for further details and notes on our evaluation of `idscrub`.
-### Models and Memory
+### Models
 * Only Spacy's `en_core_web_trf` and no Hugging Face models have been formally evaluated.
 * We therefore recommend that the current default `en_core_web_trf` is used for name scrubbing. **Other models need to be evaluated by the user.**
-> [!IMPORTANT]
-> Spacy and Hugging Face models have high memory requirements. To avoid memory-related errors. Clear the auto-generated `huggingface` folder if not in use. Do not push the `huggingface` folder (or user-defined equivalent) to GitHub.
 ## Similar Python packages
-* Similar packages exist for undertaking this task, such as [presidio](https://microsoft.github.io/presidio/), [scrubadub](https://github.com/LeapBeyond/scrubadub) and [sanityze](https://github.com/UBC-MDS/sanityze).
-* Development of `idscrub` was undertaken to: bring together different scrubbing methods across the department, adhere to infrastructure requirements, guarantee future stability and maintainability, and encourage future scrubbing methods to be added collaboratively and transparently.
-* To leverage the power of other packages, we have added methods that allow you to interact with them. These include: `IDScrub.presidio()` and `IDScrub.google_phone_numbers()`. See the [usage example notebook](https://github.com/uktrade/idscrub/blob/main/notebooks/basic_usage.ipynb) and method docstrings for further information.
-## Installation
-`idscrub` can be installed using `pip` into a Python **>=3.12** environment. Example:
-```console
-pip install idscrub
-```
-or with the spaCy transformer model (`en_core_web_trf`) already installed:
-```console
-pip instll idscrub[trf]
-```
-## How to use the code
-Basic usage example (see `notebooks/basic_usage.ipynb` for further examples):
+* Similar packages exist for undertaking this task, such as [Presidio](https://microsoft.github.io/presidio/), [Scrubadub](https://github.com/LeapBeyond/scrubadub) and [Sanityze](https://github.com/UBC-MDS/sanityze).
+* Development of `idscrub` was undertaken to:
-```python
-from idscrub import IDScrub
-scrub = IDScrub(['Our names are Hamish McDonald, L. Salah, and Elena Suárez.', 'My number is +441111111111 and I live at AA11 1AA.'])
-scrubbed_texts = scrub.scrub(scrub_methods=['spacy_persons', 'uk_phone_numbers', 'uk_postcodes'])
-print(scrubbed_texts)
-# Output: ['Our names are [PERSON], [PERSON], and [PERSON].', 'My number is [PHONENO] and I live at [POSTCODE].']
-```
+    * Bring together different scrubbing methods across the Department for Business and Trade.
+    * Adhere to infrastructure requirements.
+    * Guarantee future stability and maintainability.
+    * Encourage future scrubbing methods to be added collaboratively and transparently.
+    * Allow for full flexibility depending on the use case and required outputs.
+* To leverage the power of other packages, we have added methods that allow you to interact with them. These include: `IDScrub.presidio()` and `IDScrub.google_phone_numbers()`. See the [usage example notebook](https://github.com/uktrade/idscrub/blob/main/notebooks/basic_usage.ipynb) and method docstrings for further information.
-## AI Declaration
+## AI declaration
 AI has been used in the development of `idscrub`, primarily to develop regular expressions, suggest code refinements and draft documentation.

{idscrub-0.2.1 → idscrub-0.2.2}/idscrub.egg-info/PKG-INFO RENAMED Viewed

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: idscrub
-Version: 0.2.1
+Version: 0.2.2
 Author: Department for Business and Trade
 Requires-Python: >=3.12
 Description-Content-Type: text/markdown
@@ -21,36 +21,45 @@ Dynamic: license-file
 # idscrub 🧽✨
-## Project Information
+* Names and other personally identifying information are often present in text.
+* This information may need to be removed prior to further analysis in many cases.
+* `idscrub` identifies and removes (*✨scrubs✨*) personal data from text using [regular expressions](https://en.wikipedia.org/wiki/Regular_expression) and [named-entity recognition](https://en.wikipedia.org/wiki/Named-entity_recognition).
-* This package removes (*✨scrubs✨*) identifying personal data from text using [regular expressions](https://en.wikipedia.org/wiki/Regular_expression) and [named-entity recognition](https://en.wikipedia.org/wiki/Named-entity_recognition).
+## Installation
-> [!WARNING]
-> You must follow [GDPR guidance](https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/the-research-provisions/principles-and-grounds-for-processing/) when processing personal data using this package.
->
-> Specifically, you must:
->
-> - **Update privacy notices**: Clearly state this processing activity in new or existing privacy notices before using the package.
-> - **Ensure secure deletion**: Remove any temporary or intermediary files and outputs in a secure manner.
-> - **Ensure data subject rights upheld**: Ensure individuals can access, correct, or erase their data as required.
-> - **Maintain processing records**: Document how personal data is handled and for what purpose.
+`idscrub` can be installed using `pip` into a Python **>=3.12** environment. Example:
-### Description
+```console
+pip install idscrub
+```
+or with the spaCy transformer model (`en_core_web_trf`) already installed:
-* Names and other personally identifying information are often present in text.
-* This information may need to be removed prior to further analysis in many cases.
-* `idscrub` provides a standardised way to do this in the Department for Business and Trade.
+```console
+pip install idscrub[trf]
+```
+## How to use the code
-### Expected Outputs
+Basic usage example (see [basic_usage.ipynb](https://github.com/uktrade/idscrub/blob/main/notebooks/basic_usage.ipynb) for further examples):
-* A list of text with names and other identifying information removed.
+```python
+from idscrub import IDScrub
-> [!WARNING]
-> * This package has been designed as a *first pass* for standardised personal data removal.
-> * Users are encouraged to check and confirm outputs and conduct manual reviews where necessary, e.g. when cleaning high risk datasets.
-> * It is up to the user to assess whether this removal process needs to be supplemented by other methods for their given dataset and security requirements.
+scrub = IDScrub(['Our names are Hamish McDonald, L. Salah, and Elena Suárez.', 'My number is +441111111111 and I live at AA11 1AA.'])x
+scrubbed_texts = scrub.scrub(scrub_methods=['spacy_persons', 'uk_phone_numbers', 'uk_postcodes'])
-### Data
+print(scrubbed_texts)
+# Output: ['Our names are [PERSON], [PERSON], and [PERSON].', 'My number is [PHONENO] and I live at [POSTCODE].']
+```
+## Considerations before use
+- You must follow [GDPR guidance](https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/the-research-provisions/principles-and-grounds-for-processing/) when processing personal data using this package.
+- This package has been designed as a *first pass* for standardised personal data removal.
+- Users are encouraged to check and confirm outputs and conduct manual reviews where necessary, e.g. when cleaning high risk datasets.
+- It is up to the user to assess whether this removal process needs to be supplemented by other methods for their given dataset and security requirements.
+### Input data
 - This package is designed for text-based documents structured as a list of strings.
 - It performs best when contextual meaning can be inferred from the text.
@@ -68,50 +77,25 @@ Dynamic: license-file
 > [!IMPORTANT]
 > * See [our wiki](https://github.com/uktrade/idscrub/wiki/Evaluation) for further details and notes on our evaluation of `idscrub`.
-### Models and Memory
+### Models
 * Only Spacy's `en_core_web_trf` and no Hugging Face models have been formally evaluated.
 * We therefore recommend that the current default `en_core_web_trf` is used for name scrubbing. **Other models need to be evaluated by the user.**
-> [!IMPORTANT]
-> Spacy and Hugging Face models have high memory requirements. To avoid memory-related errors. Clear the auto-generated `huggingface` folder if not in use. Do not push the `huggingface` folder (or user-defined equivalent) to GitHub.
 ## Similar Python packages
-* Similar packages exist for undertaking this task, such as [presidio](https://microsoft.github.io/presidio/), [scrubadub](https://github.com/LeapBeyond/scrubadub) and [sanityze](https://github.com/UBC-MDS/sanityze).
-* Development of `idscrub` was undertaken to: bring together different scrubbing methods across the department, adhere to infrastructure requirements, guarantee future stability and maintainability, and encourage future scrubbing methods to be added collaboratively and transparently.
-* To leverage the power of other packages, we have added methods that allow you to interact with them. These include: `IDScrub.presidio()` and `IDScrub.google_phone_numbers()`. See the [usage example notebook](https://github.com/uktrade/idscrub/blob/main/notebooks/basic_usage.ipynb) and method docstrings for further information.
-## Installation
-`idscrub` can be installed using `pip` into a Python **>=3.12** environment. Example:
-```console
-pip install idscrub
-```
-or with the spaCy transformer model (`en_core_web_trf`) already installed:
-```console
-pip instll idscrub[trf]
-```
-## How to use the code
-Basic usage example (see `notebooks/basic_usage.ipynb` for further examples):
+* Similar packages exist for undertaking this task, such as [Presidio](https://microsoft.github.io/presidio/), [Scrubadub](https://github.com/LeapBeyond/scrubadub) and [Sanityze](https://github.com/UBC-MDS/sanityze).
+* Development of `idscrub` was undertaken to:
-```python
-from idscrub import IDScrub
-scrub = IDScrub(['Our names are Hamish McDonald, L. Salah, and Elena Suárez.', 'My number is +441111111111 and I live at AA11 1AA.'])
-scrubbed_texts = scrub.scrub(scrub_methods=['spacy_persons', 'uk_phone_numbers', 'uk_postcodes'])
-print(scrubbed_texts)
-# Output: ['Our names are [PERSON], [PERSON], and [PERSON].', 'My number is [PHONENO] and I live at [POSTCODE].']
-```
+    * Bring together different scrubbing methods across the Department for Business and Trade.
+    * Adhere to infrastructure requirements.
+    * Guarantee future stability and maintainability.
+    * Encourage future scrubbing methods to be added collaboratively and transparently.
+    * Allow for full flexibility depending on the use case and required outputs.
+* To leverage the power of other packages, we have added methods that allow you to interact with them. These include: `IDScrub.presidio()` and `IDScrub.google_phone_numbers()`. See the [usage example notebook](https://github.com/uktrade/idscrub/blob/main/notebooks/basic_usage.ipynb) and method docstrings for further information.
-## AI Declaration
+## AI declaration
 AI has been used in the development of `idscrub`, primarily to develop regular expressions, suggest code refinements and draft documentation.