DeSiMaTe 0.1.0__tar.gz

desimate-0.1.0/PKG-INFO

@@ -0,0 +1,175 @@
+Metadata-Version: 2.3
+Name: DeSiMaTe
+Version: 0.1.0
+Summary: Add your description here
+Author: Guillaume Cardoen
+Author-email: Guillaume Cardoen <Guillaume.CARDOEN@umons.ac.be>
+Requires-Dist: click>=8.5.0
+Requires-Dist: duckdb>=1.5.6
+Requires-Dist: hdbscan>=0.8.44
+Requires-Dist: joblib>=1.6.0
+Requires-Dist: numpy>=2.5.3
+Requires-Dist: pandas>=3.0.6
+Requires-Dist: pyarrow>=25.0.1
+Requires-Dist: rapidfuzz>=3.14.6
+Requires-Dist: rich>=15.0.0
+Requires-Dist: scikit-learn>=1.9.1
+Requires-Dist: scipy-stubs~=1.18.1
+Requires-Dist: sentence-transformers>=6.2.0
+Requires-Dist: torch>=2.14.1
+Requires-Dist: unidecode>=1.4.0
+Requires-Dist: wenmode>=0.15.1
+Requires-Python: >=3.14
+Description-Content-Type: text/markdown
+
+# DeSiMaTe: Detecting Similar Markdown Text
+
+[![License](https://badgen.net/pypi/license/desimate)](https://github.com/sgl-umons/desimate/blob/master/LICENSE.txt)
+[![Last release](https://badgen.net/github/release/sgl-umons/desimate/stable)](https://badgen.net/github/release/sgl-umons/desimate/stable)
+[![PyPI](https://badgen.net/pypi/v/desimate)](https://pypi.org/project/desimate)
+[![Last commit](https://badgen.net/github/last-commit/sgl-umons/desimate)](https://github.com/sgl-umons/desimate/commits/)
+
+An automated tool for detecting similar text in large collections of documents, written in Python.
+`desimate` is designed to be used as a CLI tool.
+Given a set of Markdown documents, it splits each of them into fragments (paragraphs, lists, tables, code blocks, etc.), stores them in a DuckDB database, embeds them with a learned string-embedding model, and clusters the fragments according to their similarity.
+
+The pipeline is composed of the following steps, each available as a dedicated subcommand:
+
+1. `run`: run the complete pipeline on a set of documents. It is the easiest way for quick results. Alternatively, you can use subcommands to control each step of the pipeline:
+2. `parse`: parse the given documents into fragments at a configurable granularity, and store them in a DuckDB database.
+3. `create-training-dataset`: sample values from the database into a JSON Lines file, to be used for training.
+4. `train`: train a string-embedding model (a character-level convolutional network such as NeuroSEED or CNN-ED) such that the Euclidean distance between two embeddings approximates the Levenshtein distance between the embedded strings.
+5. `embed`: embed the fragments with the trained model (syntactic embedding), or with a given sentence transformer (semantic embedding), and store the results in the database.
+6. `cluster`: group the fragments into clusters of near-duplicates, either directly with the Levenshtein distance (small datasets) or with a grid-based agglomerative clustering that uses the embeddings as a cheap first filter and the Levenshtein distance to refine the results (large datasets).
+
+This project is developed by Guillaume Cardoen at the Software Engineering Lab of the University of Mons (Belgium).
+
+## Installation
+
+`desimate` requires a Python 3.14 or newer interpreter. The easiest way to install `desimate` is to install it from PyPI
+```
+pip install desimate
+```
+
+Another easy way to install `desimate` is via `pip` from this GitHub repository
+```
+pip install git+https://github.com/sgl-umons/desimate
+```
+
+Alternatively, you can clone this repository and install it locally
+```
+git clone https://github.com/sgl-umons/desimate
+cd desimate
+pip install . # or uv sync and uv run
+```
+
+You may wish to use `desimate` in a virtual environment
+```
+python -m venv venv
+source venv/bin/activate
+pip install desimate
+```
+
+## Usage
+
+After installation, the `desimate` command-line tool should be available in your shell. All commands accept a `--config` option (defaulting to `./config.toml`) pointing to a TOML file whose content is used as default for the command options:
+
+```
+Usage: desimate [OPTIONS] COMMAND [ARGS]...
+
+Options:
+  --config PATH    The path to a TOML configuration file.
+  --help           Show this message and exit.
+
+Commands:
+  cluster                      Cluster the values stored in the database.
+  create-training-dataset      Sample values from the database, and write
+                               them in a JSON Lines file.
+  embed                        Embed the values stored in the database (or
+                               a given input file) with a model.
+  parse                        Parse the given inputs, and store the
+                               extracted fragments in the database.
+  process                      Run the full pipeline (parse, embed and
+                               cluster) in a single command.
+  train                        Train a string-embedding model.
+```
+
+The easiest way to use desimate is through the `process` subcommand. 
+```
+Usage: desimate process [OPTIONS] [INPUTS]...
+
+  Run the full pipeline in a single command: parse the given inputs (if any is
+  given), embed the values with the given model, and cluster the values of the
+  database. The embedding step is only run if `--model` is given, and the
+  clustering step is only run if `--distance` is given.
+
+Positional arguments:
+  [INPUTS]...  The files or directories (given one or multiple times;
+               directories are traversed recursively) to parse.
+
+Options:
+  -i, --input-list FILE         A file containing a list of additional inputs
+                                (one path per line) to add to the inputs.
+  -p, --parser [markdown|text]  The parser to use at which the fragments are
+                                extracted.
+  -g, --granularity TEXT        The granularity at which the fragments are
+                                extracted.
+  -m, --model FILE              The trained model used to embed the values.
+  -f, --focus-on TEXT           Only keep the fragments of the given type (can
+                                be given multiple times); every other fragment
+                                is removed from the database. (Everything is
+                                kept is not given.
+  -o, --output FILE             The path where the cluster labels will be
+                                saved as a NumPy array.
+  -d, --distance INTEGER        The maximum distance for the clustering. When
+                                given, the clustering step is run.
+  --syntactic                   Use the syntactic embedding, i.e., a trained
+                                model (default).
+  --semantic                    Use the semantic embedding, i.e., a sentence
+                                transformer.
+  --batch-size INTEGER          The batch size used for the inference
+                                (default: 1024).
+  --db, --database FILE         The DuckDB database used throughout the
+                                pipeline.  [required]
+  --help                        Show this message and exit.
+```
+
+For Markdown, each granularity is a list of Markdown AST node types at which the traversal stops, i.e., the fragment types that will be extracted. The default configuration defines two granularities:
+- `paragraph`: fragments are paragraphs, lists and tables (default);
+- `md-line`: fragments are the smallest line-level elements (text, list items, table cells and table rows).
+
+Other granularities (`line` and `file`) are available through the `text` parser, which read the file as pure text. 
+
+
+
+### Examples
+
+As an example, the following commands will parse every Markdown file of the `all_content` directory into the `my_dataset.db` database, keeping only the `paragraph`, `table`, `list` and `code` fragments.
+
+The fragments can be parsed, embedded and clustered, with a maximum distance of 15, in a single command:
+
+```bash
+desimate process all_content --db my_dataset.db --model model.torch -d 15 -o clusters.parquet  -f paragraph -f table -f list -f code
+```
+
+Alternatively, you can use subcommands to do each step at a time.
+
+```bash
+desimate parse all_content --db my_dataset.db -f paragraph -f table -f list -f code
+```
+
+A training set can then be sampled from the database:
+
+```bash
+desimate create-training-dataset training.jsonl --count 25000 --db my_dataset.db
+```
+
+An embedding model can be trained on this dataset:
+
+```bash
+desimate train nscnn --input training.jsonl --output model.torch --n-training 20000 --n-query 5000
+```
+
+## License
+
+Distributed under [GNU Lesser General Public License v3](https://github.com/sgl-umons/desimate/blob/master/LICENSE.txt).

desimate-0.1.0/README.md

@@ -0,0 +1,151 @@
+# DeSiMaTe: Detecting Similar Markdown Text
+
+[![License](https://badgen.net/pypi/license/desimate)](https://github.com/sgl-umons/desimate/blob/master/LICENSE.txt)
+[![Last release](https://badgen.net/github/release/sgl-umons/desimate/stable)](https://badgen.net/github/release/sgl-umons/desimate/stable)
+[![PyPI](https://badgen.net/pypi/v/desimate)](https://pypi.org/project/desimate)
+[![Last commit](https://badgen.net/github/last-commit/sgl-umons/desimate)](https://github.com/sgl-umons/desimate/commits/)
+
+An automated tool for detecting similar text in large collections of documents, written in Python.
+`desimate` is designed to be used as a CLI tool.
+Given a set of Markdown documents, it splits each of them into fragments (paragraphs, lists, tables, code blocks, etc.), stores them in a DuckDB database, embeds them with a learned string-embedding model, and clusters the fragments according to their similarity.
+
+The pipeline is composed of the following steps, each available as a dedicated subcommand:
+
+1. `run`: run the complete pipeline on a set of documents. It is the easiest way for quick results. Alternatively, you can use subcommands to control each step of the pipeline:
+2. `parse`: parse the given documents into fragments at a configurable granularity, and store them in a DuckDB database.
+3. `create-training-dataset`: sample values from the database into a JSON Lines file, to be used for training.
+4. `train`: train a string-embedding model (a character-level convolutional network such as NeuroSEED or CNN-ED) such that the Euclidean distance between two embeddings approximates the Levenshtein distance between the embedded strings.
+5. `embed`: embed the fragments with the trained model (syntactic embedding), or with a given sentence transformer (semantic embedding), and store the results in the database.
+6. `cluster`: group the fragments into clusters of near-duplicates, either directly with the Levenshtein distance (small datasets) or with a grid-based agglomerative clustering that uses the embeddings as a cheap first filter and the Levenshtein distance to refine the results (large datasets).
+
+This project is developed by Guillaume Cardoen at the Software Engineering Lab of the University of Mons (Belgium).
+
+## Installation
+
+`desimate` requires a Python 3.14 or newer interpreter. The easiest way to install `desimate` is to install it from PyPI
+```
+pip install desimate
+```
+
+Another easy way to install `desimate` is via `pip` from this GitHub repository
+```
+pip install git+https://github.com/sgl-umons/desimate
+```
+
+Alternatively, you can clone this repository and install it locally
+```
+git clone https://github.com/sgl-umons/desimate
+cd desimate
+pip install . # or uv sync and uv run
+```
+
+You may wish to use `desimate` in a virtual environment
+```
+python -m venv venv
+source venv/bin/activate
+pip install desimate
+```
+
+## Usage
+
+After installation, the `desimate` command-line tool should be available in your shell. All commands accept a `--config` option (defaulting to `./config.toml`) pointing to a TOML file whose content is used as default for the command options:
+
+```
+Usage: desimate [OPTIONS] COMMAND [ARGS]...
+
+Options:
+  --config PATH    The path to a TOML configuration file.
+  --help           Show this message and exit.
+
+Commands:
+  cluster                      Cluster the values stored in the database.
+  create-training-dataset      Sample values from the database, and write
+                               them in a JSON Lines file.
+  embed                        Embed the values stored in the database (or
+                               a given input file) with a model.
+  parse                        Parse the given inputs, and store the
+                               extracted fragments in the database.
+  process                      Run the full pipeline (parse, embed and
+                               cluster) in a single command.
+  train                        Train a string-embedding model.
+```
+
+The easiest way to use desimate is through the `process` subcommand. 
+```
+Usage: desimate process [OPTIONS] [INPUTS]...
+
+  Run the full pipeline in a single command: parse the given inputs (if any is
+  given), embed the values with the given model, and cluster the values of the
+  database. The embedding step is only run if `--model` is given, and the
+  clustering step is only run if `--distance` is given.
+
+Positional arguments:
+  [INPUTS]...  The files or directories (given one or multiple times;
+               directories are traversed recursively) to parse.
+
+Options:
+  -i, --input-list FILE         A file containing a list of additional inputs
+                                (one path per line) to add to the inputs.
+  -p, --parser [markdown|text]  The parser to use at which the fragments are
+                                extracted.
+  -g, --granularity TEXT        The granularity at which the fragments are
+                                extracted.
+  -m, --model FILE              The trained model used to embed the values.
+  -f, --focus-on TEXT           Only keep the fragments of the given type (can
+                                be given multiple times); every other fragment
+                                is removed from the database. (Everything is
+                                kept is not given.
+  -o, --output FILE             The path where the cluster labels will be
+                                saved as a NumPy array.
+  -d, --distance INTEGER        The maximum distance for the clustering. When
+                                given, the clustering step is run.
+  --syntactic                   Use the syntactic embedding, i.e., a trained
+                                model (default).
+  --semantic                    Use the semantic embedding, i.e., a sentence
+                                transformer.
+  --batch-size INTEGER          The batch size used for the inference
+                                (default: 1024).
+  --db, --database FILE         The DuckDB database used throughout the
+                                pipeline.  [required]
+  --help                        Show this message and exit.
+```
+
+For Markdown, each granularity is a list of Markdown AST node types at which the traversal stops, i.e., the fragment types that will be extracted. The default configuration defines two granularities:
+- `paragraph`: fragments are paragraphs, lists and tables (default);
+- `md-line`: fragments are the smallest line-level elements (text, list items, table cells and table rows).
+
+Other granularities (`line` and `file`) are available through the `text` parser, which read the file as pure text. 
+
+
+
+### Examples
+
+As an example, the following commands will parse every Markdown file of the `all_content` directory into the `my_dataset.db` database, keeping only the `paragraph`, `table`, `list` and `code` fragments.
+
+The fragments can be parsed, embedded and clustered, with a maximum distance of 15, in a single command:
+
+```bash
+desimate process all_content --db my_dataset.db --model model.torch -d 15 -o clusters.parquet  -f paragraph -f table -f list -f code
+```
+
+Alternatively, you can use subcommands to do each step at a time.
+
+```bash
+desimate parse all_content --db my_dataset.db -f paragraph -f table -f list -f code
+```
+
+A training set can then be sampled from the database:
+
+```bash
+desimate create-training-dataset training.jsonl --count 25000 --db my_dataset.db
+```
+
+An embedding model can be trained on this dataset:
+
+```bash
+desimate train nscnn --input training.jsonl --output model.torch --n-training 20000 --n-query 5000
+```
+
+## License
+
+Distributed under [GNU Lesser General Public License v3](https://github.com/sgl-umons/desimate/blob/master/LICENSE.txt).

desimate-0.1.0/pyproject.toml

@@ -0,0 +1,37 @@
+[project]
+name = "DeSiMaTe"
+version = "0.1.0"
+description = "Add your description here"
+readme = "README.md"
+requires-python = ">=3.14"
+dependencies = [
+    "click>=8.5.0",
+    "duckdb>=1.5.6",
+    "hdbscan>=0.8.44",
+    "joblib>=1.6.0",
+    "numpy>=2.5.3",
+    "pandas>=3.0.6",
+    "pyarrow>=25.0.1",
+    "rapidfuzz>=3.14.6",
+    "rich>=15.0.0",
+    "scikit-learn>=1.9.1",
+    "scipy-stubs~=1.18.1",
+    "sentence-transformers>=6.2.0",
+    "torch>=2.14.1",
+    "unidecode>=1.4.0",
+    "wenmode>=0.15.1",
+]
+
+[[project.authors]]
+name = "Guillaume Cardoen"
+email = "Guillaume.CARDOEN@umons.ac.be"
+
+[project.scripts]
+desimate = "desimate:main.cli"
+
+[build-system]
+requires = ["uv_build>=0.12.19,<0.13.0"]
+build-backend = "uv_build"
+
+[dependency-groups]
+dev = []

desimate-0.1.0/pyproject.toml.orig

@@ -0,0 +1,36 @@
+[project]
+name = "DeSiMaTe"
+version = "0.1.0"
+description = "Add your description here"
+readme = "README.md"
+authors = [
+    { name = "Guillaume Cardoen", email = "Guillaume.CARDOEN@umons.ac.be" }
+]
+requires-python = ">=3.14"
+dependencies = [
+    "click>=8.5.0",
+    "duckdb>=1.5.6",
+    "hdbscan>=0.8.44",
+    "joblib>=1.6.0",
+    "numpy>=2.5.3",
+    "pandas>=3.0.6",
+    "pyarrow>=25.0.1",
+    "rapidfuzz>=3.14.6",
+    "rich>=15.0.0",
+    "scikit-learn>=1.9.1",
+    "scipy-stubs~=1.18.1",
+    "sentence-transformers>=6.2.0",
+    "torch>=2.14.1",
+    "unidecode>=1.4.0",
+    "wenmode>=0.15.1",
+]
+
+[project.scripts]
+desimate = "desimate:main.cli"
+
+[build-system]
+requires = ["uv_build>=0.12.19,<0.13.0"]
+build-backend = "uv_build"
+
+[dependency-groups]
+dev = []

desimate-0.1.0/src/desimate/__init__.py

@@ -0,0 +1,77 @@
+from abc import ABC, abstractmethod
+
+from rich.console import Group
+from rich.live import Live
+
+from desimate.config import config
+from desimate.utils.console import TaskSpeed, get_progress
+
+
+class Step(ABC):
+    """
+    Generic step class representing a step in the detection process.
+    """
+
+    def __init__(self) -> None:
+        pass
+
+    def enter(self) -> None:
+        pass
+
+    def execute(self, *args, **kwargs):
+        self.enter()
+        r = self.run(*args, **kwargs)
+        self.exit()
+        return r
+
+    @abstractmethod
+    def run(self, *args, **kwargs) -> None:
+        """
+        Create and add every needed table to the database.
+        Run the step, and flush every table.
+        """
+
+    def exit(self) -> None:
+        pass
+
+    def __call__(self, *args, **kwargs):
+        return self.execute(*args, **kwargs)
+
+    def get_config(self):
+        return config
+
+
+class ProgressStep(Step, ABC):
+    def __init__(self, show_progress: bool = True):
+        super().__init__()
+        self.show_progress = show_progress
+        self.progress = get_progress()
+
+    def enter(self) -> None:
+        super().enter()
+        if self.show_progress:
+            self.progress.start()
+
+    def exit(self) -> None:
+        super().exit()
+        if self.show_progress:
+            self.progress.stop()
+
+
+class StatusProgressStep(ProgressStep, ABC):
+    def __init__(self, show_progress: bool = True):
+        super().__init__(show_progress)
+        self.live = Live(Group(self.progress), refresh_per_second=10)
+
+    def enter(self) -> None:
+        # super().enter()
+        if self.show_progress:
+            self.live.start()
+
+    def update_status(self, status: str):
+        self.live.update(Group(status, self.progress))
+
+    def exit(self) -> None:
+        # super().exit()
+        if self.show_progress:
+            self.live.stop()

desimate-0.1.0/src/desimate/charencoders/__init__.py

@@ -0,0 +1,38 @@
+from abc import abstractmethod
+
+from desimate import Step
+import numpy as np
+
+
+class CharacterEncoder(Step):
+    """
+    This class represents the logic for transforming a single character
+    into a vector representing this character.
+    Possible representation might include onehot encoding, UTF-8 padded to
+    4-bytes, etc.
+
+    Before using this encoder, you should first call `train` with your data.
+    You can then use `run` to encode your string or list of character.
+    """
+
+    def __init__(self, max_length: int | None = None):
+        super().__init__()
+        self.max_length = max_length
+
+    @abstractmethod
+    def train(self, texts: list[str]):
+        pass
+
+    @abstractmethod
+    def run(self, text: str | list[str]) -> np.ndarray:
+        """
+        Vectorize a str (or list of characters) towards a representative
+        vectorial embedding.
+
+        Returns a 2D array where the embeddings are in the same order of the input.
+        """
+        pass
+
+    @abstractmethod
+    def __len__(self):
+        raise NotImplementedError()

desimate-0.1.0/src/desimate/charencoders/hashing.py

@@ -0,0 +1,70 @@
+from functools import lru_cache
+
+import numpy as np
+from sklearn.feature_extraction import FeatureHasher
+from hashlib import sha256
+
+from desimate.charencoders import CharacterEncoder
+
+
+class HashingCE(CharacterEncoder):
+    """
+    Encode a character using the hashing trick.
+    """
+
+    def __init__(self, max_length: int | None = None, length: int | None = None):
+        super().__init__(max_length)
+        self.config = self.get_config()
+        self.length = length or self.config.get("length") or 128
+        if not length:
+            raise ValueError("No length was given...")
+
+    def train(self, texts: list[str]):
+        if self.max_length is None:
+            self.max_length = max((len(_t) for _t in texts))
+
+    def run(self, text: str | list[str]) -> np.ndarray:
+        feature_hasher = FeatureHasher(n_features=self.length, input_type="string")
+        encode = np.zeros((self.max_length, self.length), dtype=np.float32)
+        text_length = min(len(text), self.max_length)
+        encode[:text_length] = feature_hasher.transform(
+            [[_c] for _c in text]
+        ).toarray()[:text_length]
+        return encode
+
+    def __len__(self):
+        return self.length
+
+
+class HashingCE2(CharacterEncoder):
+    """
+    Encode a character using the hashing trick.
+    Unlike `HashingCE`, it does not rely on `FeatureHasher`,
+    but on a BLOOM filter.
+    """
+
+    def __init__(self, max_length: int | None = None, length: int = 64):
+        super().__init__(max_length)
+        self.length = length
+
+    def train(self, texts: list[str]):
+        if self.max_length is None:
+            self.max_length = max((len(_t) for _t in texts))
+
+    @lru_cache(maxsize=128)
+    def _hash(self, c):
+        digest = sha256(c.encode("utf-8")).digest()
+        h1 = int.from_bytes(digest[:8], "little") % self.length
+        h2 = int.from_bytes(digest[8:16], "little") % self.length
+        h3 = int.from_bytes(digest[16:24], "little") % self.length
+        return h1, h2, h3
+
+    def run(self, text: str | list[str]) -> np.ndarray:
+        encode = np.zeros((self.max_length, self.length), dtype=np.float32)
+        for _i, _c in enumerate(text):
+            _h1, _h2, _h3 = self._hash(_c)
+            encode[_i, [_h1, _h2, _h3]] = 1
+        return encode
+
+    def __len__(self):
+        return self.length

desimate-0.1.0/src/desimate/charencoders/onehot.py

@@ -0,0 +1,73 @@
+import logging
+
+import numpy as np
+from unidecode import unidecode
+
+from desimate.charencoders import CharacterEncoder
+
+logger = logging.getLogger(__name__)
+
+
+class OneHotCE(CharacterEncoder):
+    """
+    Encode a character using a OneHot representation.
+    An alphabet is locally maintained.
+    You should call `train` first to construct this alphabet.
+    In case a character in the text is not in the alphabet, it is
+    silently discarded.
+    """
+
+    def __init__(self, max_length: int | None = None):
+        super().__init__(max_length)
+        self.alphabet = {}
+
+    def train(self, texts: list[str]):
+        characters = set()
+        max_length = 0
+        for text in texts:
+            max_length = max(max_length, len(text))
+            characters.update(text)
+        if self.max_length is None:
+            self.max_length = max_length
+        self.alphabet = {_c: _i for _i, _c in enumerate(characters)}
+        logger.info(
+            "Total of %d characters found, forming the OneHot alphabet.",
+            len(self.alphabet),
+        )
+
+    def text_to_signal(self, text: list[str]):
+        return [self.alphabet[_c] for _c in text if _c in self.alphabet]
+
+    def run(self, text: str | list[str]) -> np.ndarray:
+        _text = text[: self.max_length]
+        if isinstance(_text, str):
+            _text = list(_text)
+        sig = self.text_to_signal(_text)
+
+        encode = np.zeros((self.max_length, len(self.alphabet)), dtype=np.float32)
+        encode[np.arange(len(sig), dtype=int), np.array(sig, dtype=int)] = 1.0
+        return encode
+
+    def __len__(self):
+        return len(self.alphabet)
+
+
+class AsciiOneHotCE(OneHotCE):
+    """
+    Encode a character using a OneHot representation.
+    To simplify the alphabet, first transform the text towards
+    ASCII only character using `unidecode`.
+
+    """
+
+    def __init__(self, max_length: int | None = None):
+        super().__init__(max_length)
+
+    def _apply(self, c):
+        return unidecode(c).lower()
+
+    def train(self, texts: list[str]):
+        super().train([self._apply(_t) for _t in texts])
+
+    def run(self, text: str | list[str]) -> np.ndarray:
+        return super().run([self._apply(_t) for _t in text])