text-mallet 0.1.0__py3-none-any.whl

text_mallet-0.1.0.dist-info/METADATA

@@ -0,0 +1,163 @@
+Metadata-Version: 2.4
+Name: text-mallet
+Version: 0.1.0
+Summary: Toolkit for Obfuscating Text Through Transformations.
+Author-email: CORAL <gallagher.eu@protonmail.com>
+License: MIT
+Project-URL: Homepage, https://text-mallet.readthedocs.io/
+Project-URL: Repository, https://github.com/DanielGall500/reef
+Project-URL: Documentation, https://text-mallet.readthedocs.io/
+Keywords: nlp,obfuscation,copyright-protection,derived text formats
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: spacy>=3.0.0
+Requires-Dist: wordfreq>=3.1.1
+Requires-Dist: pydantic>=2.13.2
+Requires-Dist: sacremoses>=0.1.1
+Requires-Dist: syntok>=1.4.4
+Requires-Dist: pandas>=3.0.1
+Requires-Dist: transformers>=5.9.0
+Requires-Dist: datasets>=4.8.5
+Requires-Dist: torch>=2.12.0
+Requires-Dist: plotnine>=0.15.4
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Requires-Dist: black>=23.0.0; extra == "dev"
+Requires-Dist: sphinx>=9.1.0; extra == "dev"
+Requires-Dist: shibuya; extra == "dev"
+Dynamic: license-file
+
+[![Powered by spaCy](https://img.shields.io/badge/Powered%20by-spaCy-09a3d5?logo=spacy&logoColor=white)](https://spacy.io)
+![English](https://img.shields.io/badge/lang-en-blue)
+![German](https://img.shields.io/badge/lang-de-blue)
+
+<br />
+<div align="center">
+  <img src="assets/mallet.svg" alt="Logo" width="200" height="200">
+
+  <p align="center">
+        Smash Text Into Obfuscated Formats
+    <br />
+    <br />
+    <br />
+    <a href="https://text-mallet.readthedocs.io/en/latest/">Documentation</a>
+    &middot;
+    <a href="https://github.com/DanielGall500/text-hammer/issues/new?labels=bug&template=bug-report---.md">Report Bug</a>
+    &middot;
+    <a href="https://github.com/DanielGall500/text-hammer/issues/new?labels=enhancement&template=feature-request---.md">Request Feature</a>
+  </p>
+</div>
+
+A lightweight text transformation engine for smashing text into [derived](https://text-plus.org/en/themen-dokumentation/atf) formats. `text-mallet` reduces the risk of privacy or copyright infringement while preserving the structural, semantic, and syntactic utility of your data for downstream NLP tasks like classification, information retrieval, topic modeling, and semantic similarity.
+
+The package natively supports text processing in **English** and **German**.
+
+## Install
+```bash
+pip install text-mallet
+
+# For obfuscating English:
+python -m spacy download en_core_web_trf
+
+# For obfuscating German:
+python -m spacy download de_dep_news_trf
+```
+
+## Strategic Obfuscation
+
+The central aim of text obfuscation using this package is to prevent **reconstructability**. Doing so involves eroding various aspects of text, such as:
+* **Word Forms**: The exact sequence of characters.
+* **Syntactic Features**: Morpho-syntactic token properties.
+* **Meanings**: Core semantic associations.
+* **Grammatical Relations**: Hierarchical sentence dependency tree structure.
+* **Sequence Information**: Flat linear sequence boundaries.
+
+Each layer contributes to the discoverability and reconstructibility of the source text. This package provides four main approaches to obfuscation: POS filtering, mutual information filtering, hierarchical scrambling, and bag-of-words. For a more detailed overview of each method, please see the [documentation](https://text-mallet.readthedocs.io/).
+
+### Why Obfuscate Text?
+When training models for text generation, we typically need all of the content and style of the original, fluent text. However, there are many tasks such as classification, semantic similarity scoring, topic modelling, and so on, where the original text may not be required in its original form to help model performance. There is typically a trove of public-domain data that can be used for model training, but there are still many questions around the usage of copyright-protected data in training. This package offers a route to preserve some of the value of copyrighted texts while hindering their reconstruction, whether that be through training-data reconstruction or model outputs.
+
+The creation of transformed texts that are thus no longer consumable by humans, but still useful for training on specific tasks.
+
+### Impact on Performance
+While training on transformed text formats can introduce minor performance drops compared to raw text baselines, `text-mallett` is explicitly designed to **accompany** non-obfuscated, public-domain collections. Introducing these anonymised, non-human-consumable formats adds proprietary text signals into your pipeline without exposing the original text to model parameters. 
+
+### Basic Obfuscation
+There are multiple general obfuscation approaches to choose from, separated into three general categories:
+* Structural Obfuscation (bag-of-words, hierarchical)
+* Part-of-Speech Filtering 
+* Mutual-Information Filtering
+
+Let's start with an example of bag-of-words scrambling.
+```python
+from tmallet import TMallet
+
+# 1. Define the Obfuscation Configuration
+algorithm = "scramble-BoW"
+config = {
+    "level": "document",
+}
+
+# 2. Define Sample Text
+sample = "Leipzig is the most populous city in the German state of Saxony. The city has a population of 633,592 residents as of 31 December 2025. It is the eighth-largest city in Germany and is part of the Central German Metropolitan Region. Leipzig is located about 150 km (90 mi) southwest of Berlin, in the southernmost part of the North German Plain (the Leipzig Bay), at the confluence of the White Elster and its tributaries Pleiße and Parthe."
+
+# 3. Load Text Mallet and Obfuscate
+tmallet = TMallet(lang="en", prefer_gpu=True)
+tmallet.load_obfuscator(algorithm, config)
+
+obfuscated_text_sample = tmallet.obfuscate(sample)
+print(obfuscated_text_sample)
+```
+
+Output
+```bash
+southernmost 150 eighth-largest most the (90 Central located Bay), Parthe. mi) in city of German Germany part population the is Plain of populous and as Leipzig the German the the km White its Metropolitan Berlin, and Leipzig in the confluence 2025. of of city It Elster is Region. December tributaries The state Saxony. (the southwest the residents the of city Leipzig German is in is Pleiße of a has at part about of 31 and North 633,592
+```
+
+**Obfuscate based on an approximation of 'word importance'**
+Mutual information measures how much information context tells you about a word.
+Words which are both _rare_ and _context-dependant_ tend to be _important_ to the meaning of a text.
+We can apply a filter to set upper or lower bounds on such an MI score, filering at the word level.
+```python
+from tmallet import TMallet
+
+algorithm = "shannon", 
+config = {
+    "threshold": 9.5,
+    "bound": "lower",
+    "replacement_mechanism": "default",
+    "max_context_length": 128,
+}
+
+tmallet = TMallet(lang=lang, prefer_gpu=True)
+tmallet.load_obfuscator(algorithm, config)
+
+text = "Data obfuscation is the process of modifying sensitive data in such a way that it is of no or little value to unauthorized intruders while still being usable by software or authorized personnel. Data masking can also be referred as anonymization, or tokenization, depending on different context."
+obfuscated = tmallet.obfuscate(text)
+print(obfuscated)
+```
+
+Output
+```
+_ _ is the _ of _ _ _ in _ a _ that it is of no or little _ to _ _ _ _ _ _ by software or _ _. _ _ can also be referred as _, or _, _ on different _.
+```
+
+If we obfuscate too strongly using mutual information, we'll end up with obfuscated sentences like:
+```
+, . . - ., or . the / . the . - –, . -
+```
+That's, well, probably not very useful. Ideally, we can find a balance between the obfuscation of some words and inclusion of others. Using the bound as `lower` instead of `upper` will preserve those words which tend to be more meaningful in a text. Running the above example again with the lower bound of 9, we get an obfuscation which preserves core semantic value while make the text more difficult to reconstruct:
+```
+Data obfuscation _ _ process _ modifying sensitive data _ such _ way _ _ _ _ _ _ _ value _ unauthorized intruders while still being usable _ _ _ authorized personnel _ Data masking _ _ _ _ _ anonymization _ _ tokenization _ depending _ _ context _
+```
+Here's an overview of an approximation of pointwise word-level mutual information, i.e. PMI(word; context), over 12,000 tokens taken from 10 random texts in the FineWeb-Edu dataset, for instance.
+![Distribution of Word-Level Mutual Information](assets/mutual-info-distribution.png)
+
+#### Contributions
+Contributions are always welcome, this package was primarily developed in conjunction with carrying out research on the potential value of such obfuscation formats in encoder pre-training. Therefore, the package has many ways it could be improved upon and there are likely opportunities for efficiency improvements in particular.
+
+#### Acknowledgements
+Part of this work was conducted within the [CORAL project](https://coral-nlp.github.io) funded by the German Federal Ministry of Research, Technology, and Space (BMFTR) under the grant number 16IS24077A. Responsibility for the content of this publication lies with the authors.

text_mallet-0.1.0.dist-info/RECORD

@@ -0,0 +1,28 @@
+text_mallet-0.1.0.dist-info/licenses/LICENSE,sha256=FDzyLfdTplJkQ0PoTuGRwyBUOpZZ-IA04Ox2X5aHQLM,1107
+tmallet/__init__.py,sha256=usScnGjg3_WbsWgYJxfRhDKE7KV0dgZr9rb9Wm7IN6w,65
+tmallet/core/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+tmallet/core/pipeline.py,sha256=ARNcGUI8k1o4Bao4MqUxVQoZTna_GgFxMxKm-bWnDq0,12075
+tmallet/obfuscators/__init__.py,sha256=FtrtALbi9bQM1lLOQtt1fe5QFNSBeAPAfcQsuk36xnk,943
+tmallet/obfuscators/base.py,sha256=xTkH5AHxKS-AMUvApLpFAEz5m-P0ICrnVI_wpH8mBiE,573
+tmallet/obfuscators/replacement_token.py,sha256=fkdvpBwQ4zV_tzF-7HSb0a39O1Eeb75J0f0H1esBQUk,486
+tmallet/obfuscators/morph/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+tmallet/obfuscators/morph/lemmatizer.py,sha256=dTphm3Mqr6TVIARQCtQE4uBAEFFiXl9VugnKxk9LEWA,463
+tmallet/obfuscators/pos/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+tmallet/obfuscators/pos/config.py,sha256=AsqBJeSBn8qVH9EKXV3HJM8k_RdOSlKpVZ1JU54-zTs,709
+tmallet/obfuscators/pos/pos_filter.py,sha256=IZTqp3tx2DowfTwFEVoUSsHfBjaf4BykMQIwx4Gjap0,8463
+tmallet/obfuscators/shannon/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+tmallet/obfuscators/shannon/config.py,sha256=br5jYkALcz8NDzQ7vD124ppktILWsVLHkfiLlFfL5oc,310
+tmallet/obfuscators/shannon/shannon_bert.py,sha256=KjOteTlHYiwZ8cnPqZhEu731Uo6o-pvMpOHALjNsspQ,11425
+tmallet/obfuscators/shannon/shannon_filter.py,sha256=swIk-10iFMwH_gLczpz4xs52UZNVwQeQ7d8mf7Q870M,7638
+tmallet/obfuscators/shannon/visualise.py,sha256=Rb2zQPAA5k3OUjdw5_qNhoKO05lMQFe6TqYWu0ZrKRE,7068
+tmallet/obfuscators/structural/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+tmallet/obfuscators/structural/config.py,sha256=r8lR8ru-7rk697HSbu_J_IC08kp0YYDPIB6fui2xqPU,314
+tmallet/obfuscators/structural/scramble_hier.py,sha256=TnwBfzB_UBEfEE5cwsKboBSuNKG2BN9XO7ZyRRF8DRE,6064
+tmallet/obfuscators/structural/scramble_linear.py,sha256=jAw3NpVO-6XNFUP717zHvn0vryMTgNCyS04gUuDtvog,3175
+tmallet/utils/__init__.py,sha256=JBSgvXwfvxwyTF_J_3cnqFkY4tSo1wiQS8E1pu39MI8,202
+tmallet/utils/helper.py,sha256=ezNBKOp9ejA4fBNwnEVLeBqg0KM02953v5VTLD2-9UI,2094
+tmallet/utils/spacy_registry.py,sha256=WSgw1zlW56CqhrMQ29R7rbykQ-wPa3KjrUcivUqnl7E,2492
+text_mallet-0.1.0.dist-info/METADATA,sha256=1Gb-jy8f7MKae7Og6hBnUtr0UCI7spW8yWGkX3_-XhY,9181
+text_mallet-0.1.0.dist-info/WHEEL,sha256=aeYiig01lYGDzBgS8HxWXOg3uV61G9ijOsup-k9o1sk,91
+text_mallet-0.1.0.dist-info/top_level.txt,sha256=trxVebx9xwb6N6JkzEmOBAyBjqslgtY2VIF-f_ZWjT0,8
+text_mallet-0.1.0.dist-info/RECORD,,

text_mallet-0.1.0.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (82.0.1)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

text_mallet-0.1.0.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Institute for Applied Informatics (InfAI), Leipzig
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

text_mallet-0.1.0.dist-info/top_level.txt

@@ -0,0 +1 @@
+tmallet

tmallet/__init__.py

@@ -0,0 +1,3 @@
+from tmallet.core.pipeline import TMallet
+
+__all__ = ["TMallet"]

tmallet/core/pipeline.py

@@ -0,0 +1,314 @@
+import os
+from functools import partial
+from itertools import islice
+from pathlib import Path
+from typing import Literal
+
+from datasets import Dataset, concatenate_datasets, load_dataset, load_from_disk
+
+from tmallet.obfuscators import (
+    HierarchicalScrambleConfig,
+    HierarchicalScrambleObfuscator,
+    LemmaObfuscator,
+    LinearScrambleConfig,
+    LinearScrambleObfuscator,
+    POSFilter,
+    POSFilterConfig,
+    ShannonFilter,
+    ShannonFilterConfig,
+)
+from tmallet.obfuscators.base import Obfuscator, SpaCyObfuscator
+from tmallet.utils import LangConfig, SpaCyInterface, flatten_dict
+
+ObfuscationTechnique = Literal[
+    "pos-filter",  # retain or remove specific POS tags
+    "scramble-hier",  # dependency-parsing structural obfuscation
+    "scramble-BoW",  # randomly shuffle words at the sentence or document level
+    "shannon",  # filter based on an approximation of word importance
+    "lemmatize",  # word-level lemmatisation - still available, but no longer supported
+]
+
+
+class TMallet:
+    """A text obfuscation manager that applies transformations to text.
+
+    This class applies selected algorithmic obfuscation techniques (such as POS filtering,
+    bag-of-words scrambling, or information-theoretic filtering) to strings, lists of text,
+    or entire datasets.
+
+    Arguments:
+        lang (LangConfig): The language configuration code (e.g., "en").
+        prefer_gpu (bool): Whether spaCy is configured to leverage GPU acceleration.
+    """
+
+    # apply_spacy_preprocessing: determines whether spacy is used or not
+    # for the initial text processing
+    # -> determined automatically based on the configuration selected
+    apply_spacy_preprocessing: bool = False
+    is_obfuscation_set_up: bool = False
+    active_obfuscator = None
+    active_config: dict | None = None
+    active_algorithm: str | None = None
+
+    def __init__(self, lang: LangConfig = "en", prefer_gpu: bool = False):
+        """Initialises the obfuscation pipeline.
+
+        Args:
+            lang (LangConfig, optional): The target language configuration - either "en" (English) or "de" (German). Defaults to "en". Let us know if you'd be interested in support for further languages.
+            prefer_gpu (bool, optional): If True, attempts to allocate spaCy operations
+                on the GPU. Defaults to False.
+        """
+        self.spacy_interface: SpaCyInterface = SpaCyInterface(
+            lang=lang, prefer_gpu=prefer_gpu
+        )
+        self.lang: LangConfig = lang
+        self.prefer_gpu = prefer_gpu
+
+    def load_obfuscator(
+        self,
+        algorithm: ObfuscationTechnique,
+        config: dict[str, str],
+    ):
+        """Validates configuration and dynamically instantiates an obfuscation algorithm.
+
+        Args:
+            algorithm (str): The identifier of the obfuscation technique (e.g., 'pos-filter').
+            config (Dict): Key-value pairings containing parameters for the specific algorithm.
+
+        Returns:
+            TMallet: The current class instance to allow for method chaining.
+        """
+        self.active_config = self._validate_config(algorithm, config)
+        self.active_obfuscator = self._get_obfuscator(algorithm)
+        self.active_obfuscator.set_config(self.active_config)
+        self.active_algorithm = algorithm
+        return self
+
+    def obfuscate(self, text: str) -> dict | str:
+        """Obfuscates standalone text strings or lists of strings.
+
+        Requires an obfuscator to be loaded via `load_obfuscator` prior to invocation.
+
+        Args:
+            text (Union[List[str], str]): Single text payload or collection of texts to process.
+
+        Raises:
+            RuntimeError: If an obfuscator and configuration have not been loaded yet.
+
+        Returns:
+            Dict: The modified, obfuscated text or collection of texts in the form of a dictionary.
+        """
+        if (
+            self.active_obfuscator is None
+            and self.active_config is None
+            and not self.active_algorithm == "lemmatize"
+        ):
+            raise RuntimeError(
+                "Please use `set_obfuscator` to setup the obfuscation details first."
+            )
+
+        if self.apply_spacy_preprocessing:
+            text = self.spacy_interface.process(text)
+
+        return self.active_obfuscator.obfuscate(text)
+
+    def _obfuscate_batch(
+        self,
+        batch: dict,
+        column: str,
+        column_obfuscated: str,
+        multi: bool = True,
+    ) -> dict:
+        """Processes a single dictionary batch extracted from a Dataset pipeline wrapper.
+
+        Args:
+            batch (Dict[str, Any]): A batch slice containing lists mapped to column keys.
+            column (str): The column key containing the raw text strings.
+            column_obfuscated (str): Target base column key for saving the output.
+            multi (bool, optional): If True, flattens a complex nested dictionary output
+                directly into the batch root elements. Defaults to True.
+
+        Raises:
+            KeyError: If the specified target data column does not exist inside the batch.
+
+        Returns:
+            Dict[str, Any]: The batch containing the obfuscated results.
+        """
+        if column not in batch.keys():
+            raise KeyError(
+                f"Invalid column provided. Please choose one of {list(batch.keys())}"
+            )
+        texts = batch[column]
+
+        if not multi:
+            batch[column_obfuscated] = [self.obfuscate(text) for text in texts]
+        else:
+            obfuscation_output = [flatten_dict(self.obfuscate(text)) for text in texts]
+
+            all_keys = obfuscation_output[0].keys()
+            batch.update(
+                {
+                    key: [sample[key] for sample in obfuscation_output]
+                    for key in all_keys
+                }
+            )
+
+        return batch
+
+    def obfuscate_dataset(
+        self,
+        dataset: Dataset,
+        column: str,
+        column_obfuscated: str,
+        batch_size: int = 10,
+        num_proc: int | None = None,
+    ):
+        """Maps obfuscation across an entire HuggingFace/compatible dataset object sequentially.
+
+        Args:
+            dataset (Dataset): The underlying dataset collection containing columns of data.
+            column (str): Key of the column containing raw target text.
+            column_obfuscated (str): Target base column key for saving the output.
+            batch_size (int, optional): Size of chunk arrays processed together. Defaults to 10.
+            num_proc (Optional[int], optional): CPU core count split handling parallel tasks.
+                Defaults to None.
+
+        Returns:
+            Dataset: A newly updated copy of the dataset containing obfuscation columns.
+        """
+        obfuscated_dataset = dataset.map(
+            partial(
+                self._obfuscate_batch,
+                column=column,
+                column_obfuscated=column_obfuscated,
+            ),
+            batched=True,
+            batch_size=batch_size,
+            desc="Obfuscating...",
+            num_proc=num_proc,
+            # cache_file_name=None,
+            load_from_cache_file=False,
+        )
+        return obfuscated_dataset
+
+    def obfuscate_dataset_by_chunk(
+        self,
+        dataset_repo: str,
+        column: str,
+        column_obfuscated: str,
+        save_chunks_to_folder: Path,
+        dataset_config: str | None = None,
+        dataset_split: str = "train",
+        chunk_size: int = 5_000,
+        batch_size: int = 100,
+        num_proc: int | None = None,
+        num_samples: int | None = None,
+    ) -> Dataset:
+        """Streams a dataset from the Hub in chunks, obfuscates each chunk,
+        and saves checkpoints to disk for fault tolerance.
+
+        Args:
+            dataset_repo (str): HuggingFace Hub repo ID or local path for load_dataset.
+            column (str): Key of the column containing raw target text.
+            column_obfuscated (str): Target base column key for saving the output.
+            save_chunks_to_folder (Path): Directory to save/load disk checkpoints.
+            dataset_config (Optional[str]): Dataset config/subset name passed to load_dataset.
+            dataset_split (str): Split to stream (e.g. "train", "validation"). Defaults to "train".
+            chunk_size (int): Number of examples per chunk. Defaults to 5_000.
+            batch_size (int): Inner batch size passed to .map. Defaults to 100.
+            num_proc (Optional[int]): CPU parallelism for .map. Defaults to None.
+            num_samples (Optional[int]): Optional cap on total examples to process.
+
+        Returns:
+            Dataset: Concatenated dataset of all processed chunks.
+        """
+        stream = load_dataset(
+            dataset_repo,
+            dataset_config,
+            split=dataset_split,
+            streaming=True,
+        )
+        if num_samples:
+            stream = stream.take(num_samples)
+
+        iterator = iter(stream)
+        processed_chunks = []
+        chunk_index = 0
+
+        while True:
+            start = chunk_index * chunk_size
+            end = start + chunk_size
+            ckpt_path = Path(save_chunks_to_folder) / f"obfuscated_ckpt_{start}_{end}"
+
+            if os.path.exists(ckpt_path):
+                print(f"Loading checkpoint {ckpt_path}")
+                chunk = load_from_disk(ckpt_path)
+                # Advance stream past already-processed examples
+                list(islice(iterator, chunk_size))
+            else:
+                rows = list(islice(iterator, chunk_size))
+                if not rows:
+                    break  # Stream exhausted
+                print(f"Processing examples {start}:{end}")
+                chunk = Dataset.from_list(rows)
+                chunk = self.obfuscate_dataset(
+                    chunk,
+                    column=column,
+                    column_obfuscated=column_obfuscated,
+                    batch_size=batch_size,
+                    num_proc=num_proc,
+                )
+                chunk.save_to_disk(ckpt_path)
+
+            processed_chunks.append(chunk)
+            chunk_index += 1
+
+        obfuscated_dataset = concatenate_datasets(processed_chunks)
+        return obfuscated_dataset
+
+    def get_active_obfuscator(self):
+        return self.active_obfuscator
+
+    def _validate_config(self, algorithm: str, config: dict):
+        match algorithm:
+            case "pos-filter":
+                return POSFilterConfig(**config)
+            case "scramble-BoW":
+                return LinearScrambleConfig(**config)
+            case "scramble-hier":
+                return HierarchicalScrambleConfig(**config)
+            case "shannon":
+                return ShannonFilterConfig(**config)
+            case "lemmatize":
+                return None
+
+    def _get_obfuscator(
+        self, algorithm: ObfuscationTechnique
+    ) -> Obfuscator | SpaCyObfuscator:
+        match algorithm:
+            case "pos-filter":
+                self.apply_spacy_preprocessing = True
+                self.spacy_interface.set_pipeline("pos")
+                return POSFilter()
+            case "scramble-hier":
+                self.apply_spacy_preprocessing = True
+                self.spacy_interface.set_pipeline("full")
+                return HierarchicalScrambleObfuscator()
+            case "scramble-BoW":
+                self.apply_spacy_preprocessing = False
+                return LinearScrambleObfuscator()
+            case "shannon":
+                self.apply_spacy_preprocessing = False
+                self.spacy_interface.set_pipeline("pos")
+                return ShannonFilter(
+                    lang=self.lang,
+                    spacy_interface=self.spacy_interface,
+                    prefer_gpu=self.prefer_gpu,
+                )
+            case "lemmatize":
+                self.apply_spacy_preprocessing = True
+                return LemmaObfuscator()
+            case _:
+                raise ValueError(
+                    f"Input {algorithm} invalid. Please provide a valid obfuscation algorithm."
+                )

tmallet/obfuscators/__init__.py

@@ -0,0 +1,25 @@
+from tmallet.obfuscators.morph.lemmatizer import LemmaObfuscator
+from tmallet.obfuscators.pos.config import POSFilterConfig
+from tmallet.obfuscators.pos.pos_filter import POSFilter
+from tmallet.obfuscators.shannon.config import ShannonFilterConfig
+from tmallet.obfuscators.shannon.shannon_bert import ShannonBERT
+from tmallet.obfuscators.shannon.shannon_filter import ShannonFilter
+from tmallet.obfuscators.structural.config import (
+    HierarchicalScrambleConfig,
+    LinearScrambleConfig,
+)
+from tmallet.obfuscators.structural.scramble_hier import HierarchicalScrambleObfuscator
+from tmallet.obfuscators.structural.scramble_linear import LinearScrambleObfuscator
+
+__all__ = [
+    "POSFilter",
+    "POSFilterConfig",
+    "ShannonFilter",
+    "ShannonFilterConfig",
+    "ShannonBERT",
+    "LinearScrambleObfuscator",
+    "LinearScrambleConfig",
+    "HierarchicalScrambleObfuscator",
+    "HierarchicalScrambleConfig",
+    "LemmaObfuscator",
+]

tmallet/obfuscators/base.py

@@ -0,0 +1,17 @@
+from spacy.tokens import Doc
+
+
+class SpaCyObfuscator:
+    def set_config(self, config):
+        raise NotImplementedError("Implement set_config for setting config details.")
+
+    def obfuscate(self, doc: Doc) -> dict | str:
+        raise NotImplementedError("SpaCy obfuscator is not implemented.")
+
+
+class Obfuscator:
+    def set_config(self, config):
+        raise NotImplementedError("Implement set_config for setting config details.")
+
+    def obfuscate(self, text: str) -> dict | str:
+        raise NotImplementedError("General-purpose obfuscator is not implemented.")

tmallet/obfuscators/morph/lemmatizer.py

@@ -0,0 +1,17 @@
+from typing import Dict
+
+from spacy.tokens import Doc
+
+from tmallet.obfuscators.base import SpaCyObfuscator
+
+
+class LemmaObfuscator(SpaCyObfuscator):
+    def obfuscate(self, doc: Doc, config: Dict = {}) -> str:
+        return self._lemmatise(doc)
+
+    def set_config(self, config: dict):
+        pass
+
+    def _lemmatise(self, doc: Doc) -> str:
+        lemmatised_text = "".join([token.lemma_ + token.whitespace_ for token in doc])
+        return lemmatised_text

tmallet/obfuscators/pos/config.py

@@ -0,0 +1,36 @@
+from enum import Enum
+from typing import List
+
+from pydantic import BaseModel
+
+
+class FilterType(str, Enum):
+    Retain = "retain"
+    Remove = "remove"
+
+
+class POSTag(str, Enum):
+    ADJ = "ADJ"
+    ADP = "ADP"
+    ADV = "ADV"
+    AUX = "AUX"
+    CCONJ = "CCONJ"
+    DET = "DET"
+    INTJ = "INTJ"
+    NOUN = "NOUN"
+    NUM = "NUM"
+    PART = "PART"
+    PRON = "PRON"
+    PROPN = "PROPN"
+    PUNCT = "PUNCT"
+    SCONJ = "SCONJ"
+    SYM = "SYM"
+    VERB = "VERB"
+    X = "X"
+    SPACE = "SPACE"
+
+
+class POSFilterConfig(BaseModel):
+    filter_type: FilterType | List[FilterType] = FilterType.Retain
+    pos_tags: List[POSTag] = [POSTag.NOUN, POSTag.PROPN]
+    replacement_mechanism: str | List[str] = "default"