PyDistintoX 0.1.0__tar.gz

pydistintox-0.1.0/PKG-INFO

@@ -0,0 +1,365 @@
+Metadata-Version: 2.3
+Name: PyDistintoX
+Version: 0.1.0
+Summary: This is a reimplementation of fundamental functionalities of 'Pydistinto', a software project developed at the [TCDH](https://tcdh.uni-trier.de/).
+Author: Leon Glüsing, Stefan Walter-Heßbrüggen
+Author-email: Leon Glüsing <leongluesing@uni-muenster.de>
+License: CC0-1.0
+Requires-Dist: altair>=6.0.0
+Requires-Dist: argparse>=1.4.0
+Requires-Dist: gensim>=4.3.3
+Requires-Dist: logging>=0.4.9.6
+Requires-Dist: numpy>=1.26.4
+Requires-Dist: pandas>=2.2.3
+Requires-Dist: pip>=25.2
+Requires-Dist: scikit-learn>=1.6.1
+Requires-Dist: scipy>=1.13.1
+Requires-Dist: spacy>=3.1
+Maintainer: Leon Glüsing
+Maintainer-email: Leon Glüsing <leongluesing@posteo.de>
+Requires-Python: >=3.11.1, <3.13
+Project-URL: Homepage of Original Project, https://www.uni-muenster.de/Wissenschaftstheorie/Forschung/PRODATPHIL/
+Project-URL: Homepage of Author, https://leongluesing.de
+Description-Content-Type: text/markdown
+
+# PyDistintoX
+
+<table style="width: 100%;">
+  <tr>
+    <td style="width: 33%; text-align: center;">
+      <img src="./docs/screenshots/overview.png" style="max-width: 100%; height: auto;"/>
+      <p><em>Overview</em></p>
+    </td>
+    <td style="width: 33%; text-align: center;">
+      <img src="./docs/screenshots/zeta_sd2.png" style="max-width: 100%; height: auto;"/>
+      <p><em>Zeta SD2</em></p>
+    </td>
+    <td style="width: 33%; text-align: center;">
+      <img src="./docs/screenshots/nfc.png" style="max-width: 100%; height: auto;"/>
+      <p><em>NFC</em></p>
+    </td>
+  </tr>
+  <tr>
+    <td colspan="3" style="text-align: center;">
+      <img src="./docs/screenshots/heatmap.png" style="max-width: 300px; height: auto;"/>
+      <p><em>Heatmap</em></p>
+    </td>
+  </tr>
+</table>
+
+
+
+
+**This is a reimplementation of fundamental functionalities of "Pydistinto", a software project developed at the [TCDH](https://tcdh.uni-trier.de/). The original script can be found [here](https://github.com/Zeta-and-Company/pydistinto).** The following major changes are worth mentioning:
+
+* using Gensim and NumPy instead of Pandas for the construction of term-document matrices; Gensim can parse corpora that do not fit into RAM, NumPy/SciPy is used to process sparse matrices with lower memory requirements,
+
+* using the inbuilt tf-idf functions of Gensim for adding more tf-idf related measures of distinctiveness (see [below](#tf-idf-measures)),
+
+This is a *lite* version of the original Pydistinto since we offer less features, f.e. no randomization of texts. However, if your corpus reaches a size that makes it difficult for the original Pydistinto to cope, it may be worth to try out our solution.
+
+PyDistintoX was developed by Leon Glüsing and Stefan Heßbrüggen-Walter. Original development was funded by the Deutsche Forschungsgemeinschaft via the project "Prodatphil: Science and Logic", project no. 537184692.
+
+---
+
+## Overview
+PyDistintoX can be used in two ways:
+- [Standalone Application (CLI)](#standalone-application): Run analyses directly from the command line.
+- [Python library](#python-library): Import functions for custom workflows.
+
+Furthermore, we support the installation via `uv` as well via `pip`.
+
+
+### Prerequisites
+Choose either:
+| uv (recommended) | Pure Python |
+|---|---|
+| install [here](https://docs.astral.sh/uv/getting-started/installation/)| >= 3.11.1, < 3.13|
+
+> **Note**: Replace `python` in the commands below with your specific version if needed.
+
+
+### Clone the Repository
+Open a terminal and clone the repository:
+   ```bash
+   git clone https://gitlab.com/leongluesing/pydistintox.git
+   cd pydistintox
+   ```
+Alternatively, you may download the code directly and go in a terminal into the project directory.
+
+
+## Standalone Application
+Run analyses directly from the command line.
+<details>
+<summary>click to expand</summary>
+
+### Installation
+
+Download spacy [model of your choice](https://spacy.io/models). You need to know the model name when using the [CLI](#command-line-options) later as well.
+
+#### uv (recommended)
+
+Install dependencies and activate virtual environment
+```bash
+uv venv
+source .venv/bin/activate
+uv pip install -e .
+```
+(use `.\.venv\Scripts\activate` on windows to activate environment)
+
+Download spacy [model of your choice](https://spacy.io/models):
+
+```bash
+uv pip install $(spacy info <MODEL_NAME> --url) 
+```
+
+On Windows use `uv pip install (spacy info <MODEL_NAME> --url)` instead.
+
+
+#### pip
+
+Create virtual environment (if it doesn't already exist)
+
+```bash
+python -m venv .venv
+source .venv/bin/activate
+```
+
+On Windows, use `.\.venv\Scripts\activate` to activate environment
+
+Now install PyDistintoX and the spaCy [model of your choice](https://spacy.io/models)
+```bash
+pip install -e .
+pip install $(spacy info <MODEL_NAME> --url)
+```
+
+---
+
+## Quickstart
+```bash
+uv run pydistintox --example
+```
+or respectively
+```bash
+python -m pydistintox --example
+```
+runs the application with texts of Arthur Conan Doyle that are part of the installation. You find them in `data/texts/example`.
+
+
+---
+
+## Usage
+
+In the example data, Doyle's detective novels are compared to his other novels in order to identify words that are distinctive for this genre. You can either:
+- place your target corpus files in `data/texts/tar` and your reference corpus files in `data/texts/ref`, **or**
+- specify custom directories via command line options (see [Command Line Options](#command-line-options) below).
+
+
+The application processes your texts in three steps:
+1. **NLP Processing:** Texts are tokenized and lemmatized. If `--save-nlp` flag is set, the results are saved as JSON in `data/interim/json`.
+2. **Statistical Analysis:** Various distinctiveness measures are calculated.
+3. **Results Visualization:** The results are automatically displayed in your default browser.
+
+If you already created the JSON files (f.e. from a previous run), you can skip the first step by setting the `--load-nlp` flag (see [Command Line Options](#command-line-options) below).
+
+You can run the application via either:
+| uv (recommended)                  | pip                                      |
+|-----------------------------------|------------------------------------------|
+| `uv run pydistintox`                | `python -m pydistintox`                         |
+
+
+### Command Line Options
+* `--debug` changes logging mode to verbose
+* `--load-nlp path/to/json/dir` skips tokenization and lemmatization, and load nlp results from path. Make sure `path/to/json/dir` contains the folders `tar` and `ref`. If no path is specified, the default `data/interim/json` is used.
+* `--example` Use the example data included with the installation to explore the program.
+* `--input-tar path/to/target/corpus` specify the directory for target corpus. Give the directory, not a path to a file! (Escape spaces in the path if necessary)
+* `--input-ref path/to/reference/corpus` specify the directory for reference corpus. Give the directory, not a path to a file! (Escape spaces in the path if necessary)
+* `--model spacy_model_name` spaCy model name in the format: `{lang}_core_{dataset}_{size}`. Example: `en_core_web_sm` (lang=en, dataset=web, size=sm). Available sizes: sm, md, lg, trf. [Find models here](https://spacy.io/models).
+* `--raw-scores` Scores will not be scaled to -1,1.
+
+</details>
+
+---
+
+
+## Python library
+Import functions for custom workflows.
+<details>
+<summary>click to expand</summary>
+
+
+### Installation
+- **If you are not using a virtual environment yet:**
+  Create one:
+  - `uv venv`
+  - `python -m venv .venv`
+
+  Activate it with:
+  - `source .venv/bin/activate` (Linux/macOS)
+  - `.\.venv\Scripts\activate` (Windows)
+
+Then run either:
+|   | uv (recommended)                  | pip                                      |
+|---|-----------------------------------|------------------------------------------|
+|Install | `uv pip install .`                | `pip install .`                         |
+|Download Model| `uv pip install $(spacy info <MODEL_NAME> --url)` | `pip install $(spacy info <MODEL_NAME> --url)` |
+|Check Installation|You can see the place of installation via `uv pip show pydistintox`.|You can see the place of installation via `pip show pydistintox`.|
+
+
+### Quickstart
+
+#### Installation
+
+```bash
+# Assuming you do not have an active virtual environment
+python3.11 -m venv .venv
+source .venv/bin/activate
+# and you are in the root directory of the pydistintox
+# project. You may install it via
+pip install .
+# You can check your installation via 
+pip show pydistintox
+
+# download spacy model
+uv pip install $(spacy info en_core_web_sm --url)
+```
+
+#### Usage
+
+
+
+```python
+from pathlib import Path
+from pydistintox import *
+
+config = Config(
+    skip_nlp = False,
+    save_nlp= JSON,
+    spacy_model_name='en_core_web_sm',
+    target=INPUT_TAR,
+    reference=INPUT_REF,
+    debug = True,
+    source='YOUR COMMENT HERE',
+    measures_to_calculate = [],  # only tf_idf_measures_to_calculate put here; for non-tf-idf not implemented yet
+    scaling = True,
+)
+non_tf_idf_measures_to_calculate = ['zeta_sd2']  # only calculate zeta_sd2
+
+...
+```
+
+**For a complete example, see the script:** [demo.py](./docs/examples/demo.py)
+
+
+### Available Import Functions
+Key functions for programmatic use (import via `from pydistintox import ...`).
+**Full documentation:** [FUNCTIONS.md](./docs/FUNCTIONS.md)
+
+</details>
+
+---
+
+## Remove PyDistintoX
+
+|   | uv                  | pip                                      |
+|---|-----------------------------------|------------------------------------------|
+|uninstall|`uv pip uninstall pydistintox`|`pip uninstall pydistintox`|
+|clean cache|`uv cache clean`|`pip cache purge`|
+|remove model|`uv pip uninstall en_core_web_sm`|`pip uninstall en_core_web_sm`|
+
+You may delete the virtual environment completely. Thereby all dependencies and downloaded models will be deleted as well: First deactivate the virtual environment (`deactivate`) and then delete its folder (`rm -r .venv` or `rmdir /s .venv` on windows).
+
+---
+
+## Troubleshooting
+
+### Installation of spaCy Model
+
+When `command not found: spacy` pops up, this means `pydistintox` is not correctly installed in your virtual environment yet (or the v.e. is not active). Try
+
+```bash
+source .venv/bin/activate
+(uv) pip install .
+(uv) pip install $(spacy info <MODEL_NAME> --url) 
+```
+
+If the installtion of a spacy model via `(uv) pip install <MODEL_NAME>` still fails, you may download the model by some other means and run `(uv) pip install path/to/model` instead.
+
+---
+
+## Technical Reference
+
+The distinctiveness measures provided are scaled to the range of [-1, 1] by default for visualization, but this scaling does not make them statistically comparable. Each measure is based on different underlying assumptions, mathematical formulations, and distributions (e.g., frequency-based vs. dispersion-based). Direct comparison of values across measures is not meaningful, unless one explicitly accounts for the differing mathematical foundations.
+
+### Non-TF-IDF Measures
+
+Using the gensim tf-idf-model as helper function, `PyDistintoX` calculates absolute and binary frequencies. This is tf-idf divided by 1, i. e. disregarding document frequency, so strictly speaking tf-no-idf. Based on this data, the following distinct measures are calculated.
+
+The following table is adapted from the foundational article on which this software is based: ["Evaluation of Measures of Distinctiveness", by Keli Du, Julia Dudar, Christof Schöch](https://doi.org/10.48694/jcls.102) ([Wayback Machine](https://web.archive.org/web/20260116101348/https://jcls.io/article/id/102/)). For more information see [here](https://zeta-project.eu/de/distinktivitaetsmasse/) ([Wayback Machine](https://web.archive.org/web/20260113084856/https://zeta-project.eu/de/distinktivitaetsmasse/))
+
+| Name                              | Type of measure      | References                                      | Evaluated in                                                                 | Implementation Key   |
+|-----------------------------------|----------------------|-------------------------------------------------|------------------------------------------------------------------------------|----------------------|
+| TF-IDF                            | Term weighting       | Luhn 1957; Spärck Jones 1972                    | Salton and Buckley 1988                                                      | –                    |
+| Ratio of relative frequencies (RRF) | Frequency-based      | Damerau 1993                                    | Gries 2010                                                                   | `rrf_dr0`            |
+| Chi-squared test (χ²)             | Frequency-based      | Dunning 1993                                    | Lijffijt et al. 2014                                                         | `chi_square_value`   |
+| Log-likelihood ratio test (LLR)  | Frequency-based      | Dunning 1993                                    | Egbert and Biber 2019; Paquot and Bestgen 2009; Lijffijt et al. 2014         | `LLR_value`          |
+| Welch’s t-test (Welch)           | Distribution-based   | Welch 1947                                      | Paquot and Bestgen 2009; Lijffijt et al. 2014                                | `welch_t_value`      |
+| Wilcoxon rank sum test (Wilcoxon) | Dispersion-based     | Wilcoxon 1945; Mann and Whitney 1947            | Paquot and Bestgen 2009; Lijffijt et al. 2014                                | `ranksumtest_value`  |
+| Burrows Zeta (Zeta_orig)          | Dispersion-based     | Burrows 2007; Craig and Kinney 2009             | Schöch 2018                                                                  | `zeta_sd0`           |
+| logarithmic Zeta (Zeta_log)       | Dispersion-based     | Schöch 2018                                     | Schöch 2018; Du et al. 2021                                                  | `zeta_sd2`           |
+| Eta                               | Dispersion-based     | Du et al. 2021                                  | Du et al. 2021                                                               | `eta_sg0`            |
+
+
+### TF-IDF Measures
+`PyDistintoX` uses the gensim tf-idf model for calculating distinctness scores. The following parameters are used:
+
+- 'nfn',
+- 'nfc',
+- 'bfn',
+- 'afn',
+- 'lfn',
+- 'ltc'
+
+For more information on the meaning of these parameters see [wikipedia](https://en.wikipedia.org/wiki/SMART_Information_Retrieval_System) [Wayback Machine](https://web.archive.org/web/20260203181537/https://en.wikipedia.org/wiki/SMART_Information_Retrieval_System), or the following paragraph:
+
+<details>
+
+<summary>
+Explanation from the Gensim Documentation
+</summary>
+
+From the [gensim documentation](https://radimrehurek.com/gensim/models/tfidfmodel.html#gensim.models.tfidfmodel.TfidfModel): 
+
+>**smartirs (str, optional) –**
+>
+>SMART (System for the Mechanical Analysis and Retrieval of Text) Information Retrieval System, a mnemonic scheme for denoting tf-idf weighting variants in the vector space model. The mnemonic for representing a combination of weights takes the form XYZ, for example ‘ntc’, ‘bpn’ and so on, where the letters represents the term weighting of the document vector.
+>
+>Term frequency weighing:
+> - b - binary,
+> - t or n - raw,
+> - a - augmented,
+> - l - logarithm,
+> - d - double logarithm,
+> - L - log average.
+>
+>Document frequency weighting:
+> - x or n - none,
+> - f - idf,
+> - t - zero-corrected idf,
+> - p - probabilistic idf.
+>
+>Document normalization:
+> - x or n - none,
+> - c - cosine,
+> - u - pivoted unique,
+> - b - pivoted character length.
+>
+>Default is ‘nfc’. For more information visit SMART Information Retrieval System.
+
+</details>
+
+### SpaCy and Gensim
+
+While gensim features its own rule-based [tokenizer](https://radimrehurek.com/gensim/utils.html#gensim.utils.simple_tokenize), we provide the opportunity to plug-in spaCy models for tokenization and lemmatization. Results of this process are saved as json files in spaCy's own format. We then extract lemmas from these data and recast them as Gensim input documents (basically list of words forming sentences which in turn are combined in a list to form a document).
+
+