idscrub 0.1.1__tar.gz → 0.2.0__tar.gz

idscrub-0.2.0/.github/pull_request_template.md

@@ -0,0 +1,27 @@
+## What
+
+<!---
+What is this PR doing, e.g. implementations, algorithms, etc.?
+ * Set the scene - you probably have a lot of context in your head that the reader doesn't have.
+ * Explain like I'm 5 - try to make as few assumptions as possible about the reader
+ * Use pictures, screenshots, or a diagram if you can, for example https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/creating-diagrams#creating-mermaid-diagrams
+--->
+
+## Why
+
+<!---
+Why is this change happening, e.g. goals, use cases, stories, etc.?
+ * Explain what the problem was that this PR addresses.
+ * Explain why this solution was chosen, and any alternatives considered.
+ * Mention any assumptions, deliberately ignored edge-cases, or changes that are left for later.
+--->
+
+## How this has been tested
+
+- [ ] I have tested locally
+- [ ] I have added a new unit test (if appropriate)
+- [ ] Testing not required
+
+## Reviewer Checklist
+
+- [ ] I have reviewed the PR and ensured no secret values are present

{idscrub-0.1.1 → idscrub-0.2.0}/.pre-commit-config.yaml

@@ -12,7 +12,7 @@ repos:
 
 # Mandatory internal hooks
 - repo: https://github.com/uktrade/github-standards
-  rev: v1.1.2  # update periodically with pre-commit autoupdate
+  rev: v1.2.1  # update periodically with pre-commit autoupdate
   hooks:
     - id: run-security-scan
       verbose: false

idscrub-0.2.0/CODEOWNERS

@@ -0,0 +1 @@
+* @uktrade/ag-data-science

{idscrub-0.1.1 → idscrub-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: idscrub
-Version: 0.1.1
+Version: 0.2.0
 Author: Department for Business and Trade
 Requires-Python: >=3.12
 Description-Content-Type: text/markdown
@@ -20,7 +20,7 @@ Dynamic: license-file
 
 # idscrub 🧽✨
 
-## Project Info
+## Project Information
 
 * This package removes (*✨scrubs✨*) identifying personal data from text using [regular expressions](https://en.wikipedia.org/wiki/Regular_expression) and [named-entity recognition](https://en.wikipedia.org/wiki/Named-entity_recognition).
 
@@ -84,15 +84,15 @@ Dynamic: license-file
 
 ## Installation
 
-`idscrub` can be installed using `pip` into a Python **>=3.12** environment. Example (with spaCy model installed):
+`idscrub` can be installed using `pip` into a Python **>=3.12** environment. Example:
 
 ```console
-pip install 'git+ssh://git@github.com/uktrade/idscrub.git#egg=idscrub[trf]'
+pip install idscrub
 ```
-or without spaCy installed (it will be installed automatically if name cleaning methods are called):
+or with the spaCy transformer model (`en_core_web_trf`) already installed:
 
 ```console
-pip install 'git+ssh://git@github.com/uktrade/idscrub.git'
+pip instll idscrub[trf]
 ```
 
 ## How to use the code
@@ -102,12 +102,12 @@ Basic usage example (see `notebooks/basic_usage.ipynb` for further examples):
 ```python
 from idscrub import IDScrub
 
-scrub = IDScrub(['Our names are Hamish McDonald, L. Salah, and Elena Suárez.', 'My number is +441111111111 and I live at AA11 1AA, Lapland.'])
-scrubbed_texts = scrub.all()
+scrub = IDScrub(['Our names are Hamish McDonald, L. Salah, and Elena Suárez.', 'My number is +441111111111 and I live at AA11 1AA.'])
+scrubbed_texts = scrub.scrub(scrub_methods=['spacy_persons', 'uk_phone_numbers', 'uk_postcodes'])
 
 print(scrubbed_texts)
 
-# Output: ['Our names are [PERSON], [PERSON], and [PERSON].', 'My number is [PHONENO] and I live at [POSTCODE], [LOCATION].']
+# Output: ['Our names are [PERSON], [PERSON], and [PERSON].', 'My number is [PHONENO] and I live at [POSTCODE].']
 ```
 
 ## AI Declaration

{idscrub-0.1.1 → idscrub-0.2.0}/README.md

@@ -1,6 +1,6 @@
 # idscrub 🧽✨
 
-## Project Info
+## Project Information
 
 * This package removes (*✨scrubs✨*) identifying personal data from text using [regular expressions](https://en.wikipedia.org/wiki/Regular_expression) and [named-entity recognition](https://en.wikipedia.org/wiki/Named-entity_recognition).
 
@@ -64,15 +64,15 @@
 
 ## Installation
 
-`idscrub` can be installed using `pip` into a Python **>=3.12** environment. Example (with spaCy model installed):
+`idscrub` can be installed using `pip` into a Python **>=3.12** environment. Example:
 
 ```console
-pip install 'git+ssh://git@github.com/uktrade/idscrub.git#egg=idscrub[trf]'
+pip install idscrub
 ```
-or without spaCy installed (it will be installed automatically if name cleaning methods are called):
+or with the spaCy transformer model (`en_core_web_trf`) already installed:
 
 ```console
-pip install 'git+ssh://git@github.com/uktrade/idscrub.git'
+pip instll idscrub[trf]
 ```
 
 ## How to use the code
@@ -82,12 +82,12 @@ Basic usage example (see `notebooks/basic_usage.ipynb` for further examples):
 ```python
 from idscrub import IDScrub
 
-scrub = IDScrub(['Our names are Hamish McDonald, L. Salah, and Elena Suárez.', 'My number is +441111111111 and I live at AA11 1AA, Lapland.'])
-scrubbed_texts = scrub.all()
+scrub = IDScrub(['Our names are Hamish McDonald, L. Salah, and Elena Suárez.', 'My number is +441111111111 and I live at AA11 1AA.'])
+scrubbed_texts = scrub.scrub(scrub_methods=['spacy_persons', 'uk_phone_numbers', 'uk_postcodes'])
 
 print(scrubbed_texts)
 
-# Output: ['Our names are [PERSON], [PERSON], and [PERSON].', 'My number is [PHONENO] and I live at [POSTCODE], [LOCATION].']
+# Output: ['Our names are [PERSON], [PERSON], and [PERSON].', 'My number is [PHONENO] and I live at [POSTCODE].']
 ```
 
 ## AI Declaration

idscrub-0.2.0/SECURITY.md

@@ -0,0 +1,47 @@
+# Security Policy
+
+The Department for Business and Trade (DBT) supports businesses to invest, grow and export, creating jobs and opportunities across the country. Read more about what we do on [GOV.UK](https://www.gov.uk/government/organisations/department-for-business-and-trade/about).
+
+## Reporting a Vulnerability
+
+If you believe you have found a security vulnerability, please submit your report to us using [here](https://hackerone.com/2680e4cd-0436-42a5-bd2a-37fd86367276/embedded_submissions/new)
+In your report please include details of:
+
+- The website, IP or page where the vulnerability can be observed.
+- A brief description of the type of vulnerability, for example; "XSS vulnerability".
+- Steps to reproduce. These should be a benign, non-destructive, proof of concept. This helps to ensure that the report can be triaged quickly and accurately. It also reduces the likelihood of duplicate reports, or malicious exploitation of some vulnerabilities, such as sub-domain takeovers.
+
+Vulnerability reporting guidelines
+
+- Please do not share the vulnerability information beyond the owner and us, without express consent from the owner
+- Vulnerabilities reported to the HackerOne platform can be submitted without the need to create a HackerOne account. However, if you wish to be updated you should create an account
+- To submit your report, you will need to agree to the HackerOne Terms and Conditions and acknowledge that you have read their Privacy Policy and Disclosure Guidelines
+- Once you have submitted the report, it will be assessed by NCC Group within five working days, and forwarded to the affected owners as soon as possible.
+
+The DBT Cyber Team will attempt to make contact with the affected owner. However, the affected owner holds responsibility for resolving the issue.
+
+## Repository security
+
+A new organisation github policy has been created that will enforce a set of security checks we expect a repository in the uktrade account to have. This policy is enabled by default for any new repositories, however existing repositories need to have it applied to them before they can be made public. The expectation is that once enough repositories have been switched from the legacy security policy to the new security policy, this new security policy is enforced across all repositories within the uktrade account
+
+### Custom properties
+
+The uktrade account makes use of custom github properties to enforce branch protection rules and run organisation level github actions. New properties can be added by logging into github using a uktrade account and using the [custom props page](https://github.com/organizations/uktrade/settings/custom-properties)
+
+### Code scanning
+
+All uktrade repositories with the new security policy applied have CodeQL scanning enabled. Individual repositories can apply their own advanced scanning rules if required
+
+### Push protection
+
+To block known secrets being committed into github, all repositories with the new security policy applied will have push protection enabled and enforced.
+
+### Branch protection
+
+An organisation ruleset has been created to apply a minimum set of branch protection rules to each public repository in the uktrade account. These rules are to be seen as the minimum, and repository admins might decide to add additional rules to their own repositories. Organisation admins and repository admins have been added to the bypass list for this branch protection ruleset. The protection rules that will be applied to each repository are:
+
+- A PR is required for merges into the default branch (usually main)
+- At least 1 approver is required before a PR can be merged
+- Any conversations on the PR must be marked as resolved
+
+As these rules are applied as an organisation ruleset, it is not possible for repository admins to add their own rules that reduce this level of protection. As an example, a repository admin could add a ruleset that drops the required number of approvers to 0 but that would have no effect as the organisation ruleset would take precedence. They could add a ruleset that sets the number of approvers to 3, and as this is not reducing the organisation ruleset protection this would take precedence

idscrub-0.2.0/SECURITY_CHECKLIST.md

@@ -0,0 +1,117 @@
+# Security Checklist
+
+This checklist is designed to make it easier to improve the security posture of a GitHub repository.
+
+- It is mandatory for public repositories.
+- This checklist must be copied over to the root of the repository.
+- The repository steward is responsible for populating the checklist, or at least approving the related pull request.
+- Any feedback should be shared with the GitHub Security working group.
+
+## Checklist
+
+- [x] [Setup the pre-commit hook framework](#setup-the-pre-commit-hook-framework)
+- [x] [Setup custom properties on the repository](#setup-custom-properties-on-the-repository)
+- [x] [Apply the correct github security policy](#apply-the-correct-github-security-policy)
+- [x] [Ensure CODEOWNERS file exists](#ensure-codeowners-file-exists)
+- [x] [Copy the SECURITY_CHECKLIST.md file](#copy-the-security_checklistmd-file)
+- [x] [Review the GitHub CI/CD overview](#review-the-github-cicd-overview)
+- [x] [Review the GitHub Safety Tips](#review-github-safety-tips)
+- [x] [Add Steward to Repository access](#add-at-least-one-steward-to-repository-access)
+- [x] [Review and limit maintainers with admin rights to the strict minimum](#review-and-limit-maintainers-with-admin-rights-to-the-strict-minimum)
+- [x] [Review the Pull Request template](#review-pull-request-template)
+- [x] [Review the SECURITY.md policy](#review-securitymd-policy)
+
+## Setup the pre-commit hook framework
+
+Several uktrade repositories already make use of the pre-commit framework for flagging code quality issues before pushing. Even in the repositories that have the pre-commit framework installed, it is still optional for an individual engineer to either avoid configuring the commit hooks, or skipping them entirely using the `--no-verify` cli argument.
+
+As part of the go live process, each engineer making changes to the repository being reopened will be required to install the organisation wide pre-commit hooks locally. When a PR is opened, an organisation level github action will run to confirm the pre-commit hooks ran on the engineers machine and will block any PRs that have not run these hooks.
+
+Instructions have been added to the [dbt hooks repository](https://github.com/uktrade/github-standards/blob/main/README.md#usage) to provide guidance on adding these organisation wide pre-commit hooks to an individual repository
+
+## Setup custom properties on the repository
+
+A set of github tags have been created at an organisation level, these must be applied to a repository to allow organisation level github actions to run on each pull request.
+
+### Mandatory custom properties
+
+- `reusable_workflow_opt_in`: This one has to be applied and set to `true` to allow this repository to apply the correct organisation branch protection ruleset and run the necessary github workflows on each PR
+- `ddat_portfolio`: The portfolio inside DDAT this repository belongs to. If your portfolio is missing, this can be added by raising an SRE ticket.
+
+### Optional custom properties
+
+- `is_docker`: If this repository builds a docker image, this tag should be added to run docker related github workflows
+- `language`: All languages used by this repository should be selected, and github workflows will run with dedicated checks on that language.
+
+## Apply the correct github security policy
+
+To add the new security policy, follow these instructions:
+
+1. As an organisation administrator, navigate to the [security config page](https://github.com/organizations/uktrade/settings/security_products).
+1. Scroll down to the **Apply configurations** sections, and enter the name of the repository to be made public in the filter input field
+1. Use the checkbox next to the results list to select all repositories being made public, then use the **Apply configuration** button to select the **Default DBT security** configuration
+1. A confirmation modal will appear displaying a summary of the action being made. Click the apply button
+1. In the repository that has had the new policy applied, navigate to the **Advanced Security** page in the repository settings. At the top of the page there should be a banner message **Modifications to some settings have been blocked by organization administrators.**
+
+### Optional: Setup CodeQL to allow PRs from repository forks
+
+For most repositories, the default CodeQL configuration applied by the **Default DBT security** policy will be sufficient. However, this default configuration does not currently support scanning PRs raised from a fork of a repository. If your repository needs to accept PRs from a fork, you must follow these steps to switch to the advanced CodeQL setup:
+
+1. Open the GitHub settings page, and navigate to the Advanced Security section using the left hand menu
+1. Scroll down to the Code Scanning section, under the Tools sub-section there will be an item for CodeQL analysis
+1. Click the ... button next to Default setup text, then choose the Switch to advanced option from the menu
+1. On the popup, click the Disable CodeQL button. Although you are disabling CodeQL, there is still a branch protection rule in place that blocks a PR unless a CodeQL scan is detected. Disabling here will not allow PRs to be merged
+1. The GitHub online editor will open to create a new file called codeql.yml in your repo, and the contents of this file will be prefilled with the languages CodeQL has detected in your repo. You can modify the contents of this file if needed, however you must leave the workflow name as `CodeQL Advanced`
+1. Once happy with the workflow file contents, click the green Commit changes button to trigger a PR to merge this into the main branch
+1. Approve and merge the PR with this workflow file. Once merged, the CodeQL scan will perform an initial scan that can take a while but you can track the progress by viewing the Actions tab for your repository
+
+## Ensure CODEOWNERS file exists
+
+The organisation rulesets require a CODEOWNERS file to be present in the repository. If you don't already have one of these, github has produced [documentation explaining](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners) what they are and why they are used.
+
+## Copy the SECURITY_CHECKLIST.md file
+
+To allow tracking of repositories that have successfully completed the reopening process, this file must be copied to the root of your repository and each of the items in the Checklist marked as completed
+
+## Review the GitHub CI/CD overview
+
+Internal contributors to the repository should review the CI/CD overview below
+![CI/CD overview](assets/CI-CD%20pipeline.svg)
+
+## Review GitHub Safety Tips
+
+Internal contributors to the repository should review the [GitHub Safety Tips](https://uktrade.atlassian.net/wiki/x/n4AEKQE)
+
+## Add at least one steward to repository access
+
+To ensure correct governance of a repository, at least one steward must be added. This will usually be the most senior engineer on the team. To add a steward to a repository:
+
+1. Open the `Collaborators and teams` settings page. The url for this is `https://github.com/uktrade/REPO_NAME/github-standards/settings/access`
+1. Use the `Add people` button to open the people finder autocomplete box.
+1. Find and click the user who is going to be a steward
+1. On the Choose a role page, select the `Steward` role.
+1. Repeat for any additional users who are going to be a steward
+
+## Review and limit maintainers with admin rights to the strict minimum
+
+You should review who has been assigned the github `admin` role. The `write` role is sufficient to allow team members to commit changes and raise pull requests
+
+## Review Pull Request template
+
+If your repository does not already contain a pull_request_template.md file, by default you will inherit the template from this repository. If you are already using your own template, you should add this section to remind reviewers they should be ensuring no secret values are visible
+
+```
+## Reviewer Checklist
+
+- [ ] I have reviewed the PR and ensured no secret values are present
+```
+
+## Review SECURITY.md policy
+
+This repository contain the SECURITY.md file, which is inherited by all repositories in the uktrade organisation account. This file should be read and understood by the repository steward, and discussed with the team to ensure all engineers understand the tooling that has been put in place
+
+## More information
+
+For more information about GitHub security standards, please refer [to this link](https://dbis.sharepoint.com/:w:/r/sites/DDaTDirectorate/Shared%20Documents/Work%20-%20GitHub%20Security/Github%20Security%20Framework/Guidelines%20and%20Policies/GitHub%20Security%20Standards%20v0.5.docx?d=wb29cd9b99ca042deb5c0cd8d670966d9&csf=1&web=1&e=6ITbnL)
+
+For more details about the security features please refer to the [GitHub Standards](https://github.com/uktrade/github-standards) repo.

{idscrub-0.1.1 → idscrub-0.2.0}/idscrub/scrub.py

@@ -19,7 +19,7 @@ from tqdm import tqdm
 from transformers import AutoModelForTokenClassification, AutoTokenizer, pipeline
 from transformers.utils import logging as trf_logging
 
-from idscrub.locations import DOWNLOAD_DIR, PROJECT_DIR
+from idscrub.locations import DOWNLOAD_DIR
 
 # Suppress Torch FutureWarning
 # TODO: Find better way
@@ -879,10 +879,46 @@ class IDScrub:
 
         return scrub_methods.get(scrub_method, lambda: "Unknown method.")()
 
+    def scrub(self, scrub_methods: list[str] = ["all"]) -> list[str]:
+        """
+        Scrubs text using given methods (in order).
+        Uses default values for the given scrub method.
+
+        Methods available (see associated method docstring for further information):
+
+        "all", "spacy_persons", "huggingface_persons", "email_addresses", "handles",
+        "ip_addresses", "uk_phone_numbers", "google_phone_numbers", "uk_postcodes"
+        "titles", "presidio"
+
+        Example:
+
+        "email_addresses" = scrub.email_addresses()
+
+        Therefore we can call:
+
+        IDScrub.scrub(scrub_methods = ["email_addresses"])
+
+        Args:
+            scrub_method (str): string name of scrub method.
+
+        Returns:
+            list[str]: The input list of text with personal information replaced.
+
+        """
+
+        for i, scrub_method in enumerate(scrub_methods):
+            if i == len(scrub_methods) - 1:
+                self.call_scrub_method(scrub_method)
+            else:
+                self.call_scrub_method(scrub_method)
+
+        return self.cleaned_texts
+
     @staticmethod
     def dataframe(
         df: pd.DataFrame = None,
         id_col: str = None,
+        exclude_cols: list[str] = None,
         scrub_methods: list[str] = ["all"],
     ) -> tuple[pd.DataFrame, pd.DataFrame]:
         """
@@ -891,6 +927,7 @@ class IDScrub:
         Args:
             df (pd.DataFrame): A Pandas dataframe to scrub.
             id_col (str): Name of the ID column in `df`. If None, an integer index starting at 1  with the name `id` is applied.
+            exclude_cols (list): Columns to exclude from scrubbing. if None all columns are scrubbed.
             scrub_methods (list[str]): Which scrub methods to apply to the DataFrame (in order).
             These are string versions of the existing methods e.g. "all" == scrub.all() and "email_addresses" == scrub.email_addresses().
 
@@ -899,6 +936,8 @@ class IDScrub:
 
         """
 
+        assert id_col in df.columns, "`id_col` is not a column in `df`. Please check."
+
         if id_col:
             ids = df[id_col].to_list()
         if not id_col:
@@ -908,14 +947,18 @@ class IDScrub:
         assert isinstance(df, pd.DataFrame), "`df` must be a Pandas DataFrame."
         assert len(df) == len(ids), "Length of dataframe is different to the length of IDs."
 
+        if exclude_cols is None:
+            cols_to_scrub = df.columns.to_list()
+        else:
+            cols_to_scrub = [col for col in df.columns if col not in exclude_cols]
+
+        cols_to_scrub.remove(id_col)
+
         scrubbed_df = df.copy()
 
         all_scrubbed_data = []
 
-        for col in tqdm(scrubbed_df.columns):
-            if col == id_col:
-                continue
-
+        for col in tqdm(cols_to_scrub):
             original_dtype = scrubbed_df[col].dtype
             scrubbed_df[col] = scrubbed_df[col].astype(str)
 
@@ -944,4 +987,6 @@ class IDScrub:
         all_scrubbed_data = pd.concat(all_scrubbed_data).reset_index(drop=True)
         all_scrubbed_data = all_scrubbed_data.where(pd.notna(all_scrubbed_data), None)
 
+        assert df.shape == scrubbed_df.shape, "Original and scrubbed dataframe not the same shape. Check."
+
         return scrubbed_df, all_scrubbed_data

{idscrub-0.1.1 → idscrub-0.2.0}/idscrub.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: idscrub
-Version: 0.1.1
+Version: 0.2.0
 Author: Department for Business and Trade
 Requires-Python: >=3.12
 Description-Content-Type: text/markdown
@@ -20,7 +20,7 @@ Dynamic: license-file
 
 # idscrub 🧽✨
 
-## Project Info
+## Project Information
 
 * This package removes (*✨scrubs✨*) identifying personal data from text using [regular expressions](https://en.wikipedia.org/wiki/Regular_expression) and [named-entity recognition](https://en.wikipedia.org/wiki/Named-entity_recognition).
 
@@ -84,15 +84,15 @@ Dynamic: license-file
 
 ## Installation
 
-`idscrub` can be installed using `pip` into a Python **>=3.12** environment. Example (with spaCy model installed):
+`idscrub` can be installed using `pip` into a Python **>=3.12** environment. Example:
 
 ```console
-pip install 'git+ssh://git@github.com/uktrade/idscrub.git#egg=idscrub[trf]'
+pip install idscrub
 ```
-or without spaCy installed (it will be installed automatically if name cleaning methods are called):
+or with the spaCy transformer model (`en_core_web_trf`) already installed:
 
 ```console
-pip install 'git+ssh://git@github.com/uktrade/idscrub.git'
+pip instll idscrub[trf]
 ```
 
 ## How to use the code
@@ -102,12 +102,12 @@ Basic usage example (see `notebooks/basic_usage.ipynb` for further examples):
 ```python
 from idscrub import IDScrub
 
-scrub = IDScrub(['Our names are Hamish McDonald, L. Salah, and Elena Suárez.', 'My number is +441111111111 and I live at AA11 1AA, Lapland.'])
-scrubbed_texts = scrub.all()
+scrub = IDScrub(['Our names are Hamish McDonald, L. Salah, and Elena Suárez.', 'My number is +441111111111 and I live at AA11 1AA.'])
+scrubbed_texts = scrub.scrub(scrub_methods=['spacy_persons', 'uk_phone_numbers', 'uk_postcodes'])
 
 print(scrubbed_texts)
 
-# Output: ['Our names are [PERSON], [PERSON], and [PERSON].', 'My number is [PHONENO] and I live at [POSTCODE], [LOCATION].']
+# Output: ['Our names are [PERSON], [PERSON], and [PERSON].', 'My number is [PHONENO] and I live at [POSTCODE].']
 ```
 
 ## AI Declaration

{idscrub-0.1.1 → idscrub-0.2.0}/idscrub.egg-info/SOURCES.txt

@@ -1,10 +1,14 @@
 .gitignore
 .pre-commit-config.yaml
+CODEOWNERS
 LICENSE
 Makefile
 README.md
+SECURITY.md
+SECURITY_CHECKLIST.md
 pyproject.toml
 uv.lock
+.github/pull_request_template.md
 .github/workflows/cd.yml
 .github/workflows/ci.yml
 idscrub/__init__.py
@@ -26,4 +30,5 @@ test/test_log.py
 test/test_persidio.py
 test/test_phonenumbers.py
 test/test_regex.py
+test/test_scrub.py
 test/test_spacy.py

{idscrub-0.1.1 → idscrub-0.2.0}/notebooks/basic_usage.ipynb

@@ -9,22 +9,139 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": 12,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "INFO: Texts loaded.\n",
+      "INFO: Scrubbing names using SpaCy model `en_core_web_trf`...\n",
+      "100%|██████████| 2/2 [00:00<00:00, 44.29it/s]\n",
+      "INFO: 3 spacy person scrubbed.\n",
+      "INFO: Scrubbing phone numbers using regex...\n",
+      "INFO: 1 uk phone numbers scrubbed.\n",
+      "INFO: Scrubbing UK postcodes using regex...\n",
+      "INFO: 1 uk postcodes scrubbed.\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "['Our names are [PERSON], [PERSON], and [PERSON].', 'My number is [PHONENO] and I live at [POSTCODE], Lapland.']\n"
+     ]
+    }
+   ],
+   "source": [
+    "from idscrub import IDScrub\n",
+    "\n",
+    "scrub = IDScrub(\n",
+    "    [\n",
+    "        \"Our names are Hamish McDonald, L. Salah, and Elena Suárez.\",\n",
+    "        \"My number is +441111111111 and I live at AA11 1AA, Lapland.\",\n",
+    "    ]\n",
+    ")\n",
+    "\n",
+    "scrubbed_texts = scrub.scrub(scrub_methods=[\"spacy_persons\", \"uk_phone_numbers\", \"uk_postcodes\"])\n",
+    "\n",
+    "print(scrubbed_texts)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>text_id</th>\n",
+       "      <th>scrubbed_spacy_person</th>\n",
+       "      <th>scrubbed_uk_phone_numbers</th>\n",
+       "      <th>scrubbed_uk_postcodes</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>0</th>\n",
+       "      <td>1</td>\n",
+       "      <td>[Hamish McDonald, L. Salah, Elena Suárez]</td>\n",
+       "      <td>None</td>\n",
+       "      <td>None</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>1</th>\n",
+       "      <td>2</td>\n",
+       "      <td>None</td>\n",
+       "      <td>[+441111111111]</td>\n",
+       "      <td>[AA11 1AA]</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "   text_id                      scrubbed_spacy_person  \\\n",
+       "0        1  [Hamish McDonald, L. Salah, Elena Suárez]   \n",
+       "1        2                                       None   \n",
+       "\n",
+       "  scrubbed_uk_phone_numbers scrubbed_uk_postcodes  \n",
+       "0                      None                  None  \n",
+       "1           [+441111111111]            [AA11 1AA]  "
+      ]
+     },
+     "execution_count": 13,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "scrub.get_scrubbed_data()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Or scrub `all`:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
    "metadata": {},
    "outputs": [
     {
      "name": "stderr",
      "output_type": "stream",
      "text": [
-      "/Users/euansoutter/Documents/code/idscrub/.venv/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
-      "  from .autonotebook import tqdm as notebook_tqdm\n",
       "INFO: Texts loaded.\n",
       "INFO: Scrubbing using Presidio...\n",
-      "100%|██████████| 2/2 [00:00<00:00,  9.48it/s]\n",
+      "100%|██████████| 2/2 [00:00<00:00, 25.19it/s]\n",
       "INFO: 3 presidio person scrubbed.\n",
       "INFO: 1 presidio location scrubbed.\n",
       "INFO: Scrubbing names using SpaCy model `en_core_web_trf`...\n",
-      "100%|██████████| 2/2 [00:00<00:00, 55.76it/s]\n",
+      "100%|██████████| 2/2 [00:00<00:00, 48.66it/s]\n",
       "INFO: 0 spacy person scrubbed.\n",
       "INFO: Scrubbing GB phone numbers using Google's `phonenumbers`...\n",
       "INFO: 0 gb phone numbers scrubbed.\n",
@@ -59,14 +176,15 @@
     "        \"My number is +441111111111 and I live at AA11 1AA, Lapland.\",\n",
     "    ]\n",
     ")\n",
-    "scrubbed_texts = scrub.all()\n",
+    "\n",
+    "scrubbed_texts = scrub.scrub(scrub_methods=[\"all\"])\n",
     "\n",
     "print(scrubbed_texts)"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": 15,
    "metadata": {},
    "outputs": [
     {
@@ -128,7 +246,7 @@
        "1                  [Lapland]           [+441111111111]            [AA11 1AA]  "
       ]
      },
-     "execution_count": 2,
+     "execution_count": 15,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -146,7 +264,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 16,
    "metadata": {},
    "outputs": [
     {
@@ -155,7 +273,7 @@
      "text": [
       "INFO: Texts loaded.\n",
       "INFO: Scrubbing using Presidio...\n",
-      "100%|██████████| 2/2 [00:00<00:00, 25.84it/s]\n",
+      "100%|██████████| 2/2 [00:00<00:00, 23.03it/s]\n",
       "INFO: 3 presidio person scrubbed.\n",
       "INFO: Scrubbing GB phone numbers using Google's `phonenumbers`...\n",
       "INFO: 0 gb phone numbers scrubbed.\n",
@@ -206,7 +324,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": 17,
    "metadata": {},
    "outputs": [
     {
@@ -271,7 +389,7 @@
        "1              [ACHILLES]           [+441111111111]            [AA11 1AA]  "
       ]
      },
-     "execution_count": 4,
+     "execution_count": 17,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -290,7 +408,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
+   "execution_count": 18,
    "metadata": {},
    "outputs": [
     {
@@ -299,7 +417,7 @@
      "text": [
       "INFO: Texts loaded.\n",
       "INFO: Scrubbing using Presidio...\n",
-      "100%|██████████| 2/2 [00:00<00:00, 26.18it/s]\n",
+      "100%|██████████| 2/2 [00:00<00:00, 23.38it/s]\n",
       "INFO: 3 presidio person scrubbed.\n",
       "INFO: 1 presidio iban code scrubbed.\n"
      ]
@@ -325,7 +443,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": 19,
    "metadata": {},
    "outputs": [
     {
@@ -381,7 +499,7 @@
        "1    [GB91BKEN10000041610008]  "
       ]
      },
-     "execution_count": 6,
+     "execution_count": 19,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -399,7 +517,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 7,
+   "execution_count": 20,
    "metadata": {},
    "outputs": [
     {
@@ -505,7 +623,7 @@
        "4  They did not expected a reply from otis.reddin...  "
       ]
      },
-     "execution_count": 7,
+     "execution_count": 20,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -551,21 +669,21 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 8,
+   "execution_count": 21,
    "metadata": {},
    "outputs": [
     {
      "name": "stderr",
      "output_type": "stream",
      "text": [
-      "  0%|          | 0/5 [00:00<?, ?it/s]INFO: Texts loaded.\n",
+      "  0%|          | 0/3 [00:00<?, ?it/s]INFO: Texts loaded.\n",
       "INFO: Scrubbing using Presidio...\n",
-      "100%|██████████| 5/5 [00:00<00:00, 24.83it/s]\n",
+      "100%|██████████| 5/5 [00:00<00:00, 18.99it/s]\n",
       "INFO: 4 presidio person scrubbed.\n",
       "INFO: 4 presidio person scrubbed.\n",
       "INFO: 4 presidio person scrubbed.\n",
       "INFO: Scrubbing names using SpaCy model `en_core_web_trf`...\n",
-      "100%|██████████| 5/5 [00:00<00:00, 71.71it/s]\n",
+      "100%|██████████| 5/5 [00:00<00:00, 67.00it/s]\n",
       "INFO: 0 spacy person scrubbed.\n",
       "INFO: Scrubbing GB phone numbers using Google's `phonenumbers`...\n",
       "INFO: 0 gb phone numbers scrubbed.\n",
@@ -581,34 +699,13 @@
       "INFO: 0 uk postcodes scrubbed.\n",
       "INFO: Scrubbing titles using regex...\n",
       "INFO: 2 titles scrubbed.\n",
-      " 40%|████      | 2/5 [00:02<00:03,  1.25s/it]INFO: Texts loaded.\n",
+      " 33%|███▎      | 1/3 [00:03<00:06,  3.24s/it]INFO: Texts loaded.\n",
       "INFO: Scrubbing using Presidio...\n",
-      "100%|██████████| 5/5 [00:00<00:00, 29.98it/s]\n",
+      "100%|██████████| 5/5 [00:00<00:00, 21.83it/s]\n",
       "INFO: 2 presidio person scrubbed.\n",
       "INFO: 2 presidio person scrubbed.\n",
       "INFO: Scrubbing names using SpaCy model `en_core_web_trf`...\n",
-      "100%|██████████| 5/5 [00:00<00:00, 96.09it/s]\n",
-      "INFO: 0 spacy person scrubbed.\n",
-      "INFO: Scrubbing GB phone numbers using Google's `phonenumbers`...\n",
-      "INFO: 0 gb phone numbers scrubbed.\n",
-      "INFO: Scrubbing email addresses using regex...\n",
-      "INFO: 0 email addresses scrubbed.\n",
-      "INFO: Scrubbing @user handles using regex...\n",
-      "INFO: 0 handles scrubbed.\n",
-      "INFO: Scrubbing IP addresses using regex...\n",
-      "INFO: 0 ip addresses scrubbed.\n",
-      "INFO: Scrubbing phone numbers using regex...\n",
-      "INFO: 0 uk phone numbers scrubbed.\n",
-      "INFO: Scrubbing UK postcodes using regex...\n",
-      "INFO: 0 uk postcodes scrubbed.\n",
-      "INFO: Scrubbing titles using regex...\n",
-      "INFO: 0 titles scrubbed.\n",
-      " 60%|██████    | 3/5 [00:04<00:03,  1.66s/it]INFO: Texts loaded.\n",
-      "INFO: Scrubbing using Presidio...\n",
-      "100%|██████████| 5/5 [00:00<00:00, 26.73it/s]\n",
-      "INFO: 1 presidio person scrubbed.\n",
-      "INFO: Scrubbing names using SpaCy model `en_core_web_trf`...\n",
-      "100%|██████████| 5/5 [00:00<00:00, 89.71it/s]\n",
+      "100%|██████████| 5/5 [00:00<00:00, 84.69it/s]\n",
       "INFO: 0 spacy person scrubbed.\n",
       "INFO: Scrubbing GB phone numbers using Google's `phonenumbers`...\n",
       "INFO: 0 gb phone numbers scrubbed.\n",
@@ -624,15 +721,15 @@
       "INFO: 0 uk postcodes scrubbed.\n",
       "INFO: Scrubbing titles using regex...\n",
       "INFO: 0 titles scrubbed.\n",
-      " 80%|████████  | 4/5 [00:07<00:01,  1.91s/it]INFO: Texts loaded.\n",
+      " 67%|██████▋   | 2/3 [00:06<00:03,  3.24s/it]INFO: Texts loaded.\n",
       "INFO: Scrubbing using Presidio...\n",
-      "100%|██████████| 5/5 [00:00<00:00, 21.44it/s]\n",
-      "INFO: 3 presidio email address scrubbed.\n",
+      "100%|██████████| 5/5 [00:00<00:00, 29.32it/s]\n",
+      "INFO: 5 presidio url scrubbed.\n",
       "INFO: 2 presidio person scrubbed.\n",
       "INFO: 3 presidio email address scrubbed.\n",
       "INFO: 3 presidio email address scrubbed.\n",
       "INFO: Scrubbing names using SpaCy model `en_core_web_trf`...\n",
-      "100%|██████████| 5/5 [00:00<00:00, 57.46it/s]\n",
+      "100%|██████████| 5/5 [00:00<00:00, 66.37it/s]\n",
       "INFO: 0 spacy person scrubbed.\n",
       "INFO: Scrubbing GB phone numbers using Google's `phonenumbers`...\n",
       "INFO: 0 gb phone numbers scrubbed.\n",
@@ -648,7 +745,7 @@
       "INFO: 4 uk postcodes scrubbed.\n",
       "INFO: Scrubbing titles using regex...\n",
       "INFO: 0 titles scrubbed.\n",
-      "100%|██████████| 5/5 [00:09<00:00,  1.91s/it]\n"
+      "100%|██████████| 3/3 [00:08<00:00,  2.94s/it]\n"
      ]
     },
     {
@@ -685,7 +782,7 @@
        "      <td>A</td>\n",
        "      <td>[TITLE]. [PERSON] walked off; and [PERSON] rem...</td>\n",
        "      <td>To [PERSON] she is always the woman.</td>\n",
-       "      <td>My dear [PERSON], do not waste your time upon ...</td>\n",
+       "      <td>My dear Victor, do not waste your time upon th...</td>\n",
        "      <td>The letter to [EMAIL_ADDRESS] was stamped with...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
@@ -740,7 +837,7 @@
        "4  When you have eliminated the impossible, whate...   \n",
        "\n",
        "                                        Frankenstein  \\\n",
-       "0  My dear [PERSON], do not waste your time upon ...   \n",
+       "0  My dear Victor, do not waste your time upon th...   \n",
        "1  Learn from me, if not by my precepts, at least...   \n",
        "2  I had worked hard for nearly two years, for th...   \n",
        "3  Nothing is more painful to the human mind than...   \n",
@@ -754,7 +851,7 @@
        "4  They did not expected a reply from [EMAIL_ADDR...  "
       ]
      },
-     "execution_count": 8,
+     "execution_count": 21,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -762,14 +859,14 @@
    "source": [
     "from idscrub import IDScrub\n",
     "\n",
-    "scrubbed_df, scrubbed_data = IDScrub.dataframe(df=df, id_col=\"ID\", scrub_methods=[\"all\"])\n",
+    "scrubbed_df, scrubbed_data = IDScrub.dataframe(df=df, id_col=\"ID\", exclude_cols=[\"Frankenstein\"], scrub_methods=[\"all\"])\n",
     "\n",
     "scrubbed_df"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 9,
+   "execution_count": 22,
    "metadata": {},
    "outputs": [
     {
@@ -862,17 +959,6 @@
        "    <tr>\n",
        "      <th>5</th>\n",
        "      <td>A</td>\n",
-       "      <td>Frankenstein</td>\n",
-       "      <td>[Victor]</td>\n",
-       "      <td>None</td>\n",
-       "      <td>None</td>\n",
-       "      <td>None</td>\n",
-       "      <td>None</td>\n",
-       "      <td>None</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <th>6</th>\n",
-       "      <td>A</td>\n",
        "      <td>Fake book</td>\n",
        "      <td>None</td>\n",
        "      <td>None</td>\n",
@@ -882,7 +968,7 @@
        "      <td>[SW1A 2AA]</td>\n",
        "    </tr>\n",
        "    <tr>\n",
-       "      <th>7</th>\n",
+       "      <th>6</th>\n",
        "      <td>B</td>\n",
        "      <td>Fake book</td>\n",
        "      <td>[Mick Jagger, David Bowie]</td>\n",
@@ -893,7 +979,7 @@
        "      <td>[SW1A 2WH]</td>\n",
        "    </tr>\n",
        "    <tr>\n",
-       "      <th>8</th>\n",
+       "      <th>7</th>\n",
        "      <td>C</td>\n",
        "      <td>Fake book</td>\n",
        "      <td>None</td>\n",
@@ -904,7 +990,7 @@
        "      <td>[SW19 5AE]</td>\n",
        "    </tr>\n",
        "    <tr>\n",
-       "      <th>9</th>\n",
+       "      <th>8</th>\n",
        "      <td>E</td>\n",
        "      <td>Fake book</td>\n",
        "      <td>None</td>\n",
@@ -925,11 +1011,10 @@
        "2  C                Pride and Prejudice                 [Elizabeth]   \n",
        "3  A  The Adventures of Sherlock Holmes           [Sherlock Holmes]   \n",
        "4  D  The Adventures of Sherlock Holmes                    [Watson]   \n",
-       "5  A                       Frankenstein                    [Victor]   \n",
-       "6  A                          Fake book                        None   \n",
-       "7  B                          Fake book  [Mick Jagger, David Bowie]   \n",
-       "8  C                          Fake book                        None   \n",
-       "9  E                          Fake book                        None   \n",
+       "5  A                          Fake book                        None   \n",
+       "6  B                          Fake book  [Mick Jagger, David Bowie]   \n",
+       "7  C                          Fake book                        None   \n",
+       "8  E                          Fake book                        None   \n",
        "\n",
        "  scrubbed_titles  scrubbed_presidio_email_address  \\\n",
        "0            [Mr]                             None   \n",
@@ -937,11 +1022,10 @@
        "2            None                             None   \n",
        "3            None                             None   \n",
        "4            None                             None   \n",
-       "5            None                             None   \n",
-       "6            None      [freddie.mercury@queen.com]   \n",
-       "7            None                             None   \n",
-       "8            None     [serena.williams@tennis.com]   \n",
-       "9            None  [otis.redding@dockofthebay.org]   \n",
+       "5            None      [freddie.mercury@queen.com]   \n",
+       "6            None                             None   \n",
+       "7            None     [serena.williams@tennis.com]   \n",
+       "8            None  [otis.redding@dockofthebay.org]   \n",
        "\n",
        "  scrubbed_presidio_iban_code         scrubbed_presidio_url  \\\n",
        "0                        None                          None   \n",
@@ -949,11 +1033,10 @@
        "2                        None                          None   \n",
        "3                        None                          None   \n",
        "4                        None                          None   \n",
-       "5                        None                          None   \n",
-       "6    [GB91BKEN10000041610008]       [freddie.me, queen.com]   \n",
-       "7                        None                          None   \n",
-       "8                        None                  [tennis.com]   \n",
-       "9                        None  [otis.red, dockofthebay.org]   \n",
+       "5    [GB91BKEN10000041610008]       [freddie.me, queen.com]   \n",
+       "6                        None                          None   \n",
+       "7                        None                  [tennis.com]   \n",
+       "8                        None  [otis.red, dockofthebay.org]   \n",
        "\n",
        "  scrubbed_uk_postcodes  \n",
        "0                  None  \n",
@@ -961,14 +1044,13 @@
        "2                  None  \n",
        "3                  None  \n",
        "4                  None  \n",
-       "5                  None  \n",
-       "6            [SW1A 2AA]  \n",
-       "7            [SW1A 2WH]  \n",
-       "8            [SW19 5AE]  \n",
-       "9             [EH8 8DX]  "
+       "5            [SW1A 2AA]  \n",
+       "6            [SW1A 2WH]  \n",
+       "7            [SW19 5AE]  \n",
+       "8             [EH8 8DX]  "
       ]
      },
-     "execution_count": 9,
+     "execution_count": 22,
      "metadata": {},
      "output_type": "execute_result"
     }

idscrub-0.2.0/test/test_scrub.py

@@ -0,0 +1,48 @@
+import pandas as pd
+from idscrub import IDScrub
+from pandas.testing import assert_frame_equal
+
+
+# Note: These tests will fail if the kernel has not been restarted since the SpaCy model was downloaded.
+def test_scrub(scrub_object):
+    scrubbed = scrub_object.scrub(scrub_methods=["spacy_persons", "uk_phone_numbers", "uk_postcodes"])
+    assert scrubbed == [
+        "Our names are [PERSON], [PERSON], and [PERSON].",
+        "My number is [PHONENO] and I live at [POSTCODE].",
+    ]
+
+
+def test_scrub_text_id():
+    scrub = IDScrub(["Our names are Hamish McDonald, L. Salah, and Elena Suárez."] * 10)
+
+    scrub.scrub(scrub_methods=["spacy_persons"])
+
+    df = scrub.get_scrubbed_data()
+
+    assert df["text_id"].max() == 10
+    assert len(df["text_id"]) == 10
+
+
+def test_scrub_get_scrubbed_data(scrub_object):
+    scrub_object.scrub(scrub_methods=["uk_postcodes"])
+    df = scrub_object.get_scrubbed_data()
+
+    expected_df = pd.DataFrame(
+        {
+            "text_id": {0: 2},
+            "scrubbed_uk_postcodes": {0: ["AA11 1AA"]},
+        }
+    )
+
+    assert_frame_equal(df, expected_df)
+
+
+def test_scrub_order(scrub_object):
+    scrub_object.scrub(scrub_methods=["uk_postcodes", "uk_phone_numbers", "spacy_persons"])
+
+    assert scrub_object.get_scrubbed_data().columns.to_list() == [
+        "text_id",
+        "scrubbed_uk_postcodes",
+        "scrubbed_uk_phone_numbers",
+        "scrubbed_spacy_person",
+    ]