textdirectory 0.3.2__tar.gz → 0.4.0__tar.gz

textdirectory-0.4.0/.gitignore

@@ -0,0 +1,106 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+env/
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# pyenv
+.python-version
+
+# celery beat schedule file
+celerybeat-schedule
+
+# SageMath parsed files
+*.sage.py
+
+# dotenv
+.env
+
+# virtualenv
+.venv
+venv/
+ENV/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+
+# PyCharm
+.idea/
+test.txt

textdirectory-0.4.0/AUTHORS.md

@@ -0,0 +1,9 @@
+# Credits
+
+## Development Lead
+
+* Ingo Kleiber <ingo@kleiber.me>
+
+## Contributors
+
+* Lyuba Dimitrova <dimitrova@cl.uni-heidelberg.de>

textdirectory-0.4.0/CHANGELOG.md

@@ -0,0 +1,127 @@
+# Changelog
+
+## 0.4.0 (2026-07-26)
+
+* modernized the packaging: pyproject.toml + hatchling, src/ layout, uv-based development workflow
+* supported Python versions are now 3.10–3.13
+* made spaCy and psutil optional dependencies (`pip install 'textdirectory[nlp]'` plus `python -m spacy download en_core_web_sm`)
+* dropped the numpy dependency (replaced by the standard library random and statistics modules)
+* spaCy models are now loaded once and cached instead of being reloaded per file and transformation
+* added type hints throughout the package (py.typed)
+* added `--version` and `python -m textdirectory` support to the CLI
+* CLI filter arguments are now coerced to the expected types (filesize, TTR, and similarity filters previously failed from the CLI)
+* CLI errors now exit with a non-zero exit code
+* replaced Travis CI with GitHub Actions; replaced the legacy Read the Docs configuration; rewrote the documentation in Markdown
+* the test corpus is no longer shipped inside the wheel (moved to tests/data/testdata/)
+* fixed destage_transformation, which could never destage anything
+* fixed the CrudeSpellChecker correction cache (wrong cache key) and a crash on tokens without word characters
+* fixed transformation_usas_en_semtag: HTTPS, request timeout, and raising on HTTP errors
+* stage_transformation and run_filters now validate names against the actual registries
+* fixed filter_by_random_sampling turning the aggregation into a numpy array; note that the random sequence differs from previous versions
+* transformation_remove_stopwords now raises for an unknown source or missing stopwords file instead of returning False
+* tabulate_flat_list_of_dicts returns '' for an empty list; type_token_ratio returns 0.0 for empty text
+* aggregate_to_file no longer silently drops unencodable characters when writing
+
+## 0.3.4 (2022-09-25)
+
+* added transformation_replace_string
+* added filter_by_type_token_ratio
+* added tests for running transformations
+* added contractions fix to transformation_lemmatize
+* added the --disable_tqdm option to the CLI
+* added the ability to output aggregations to the console when using the CLI
+* added (static) coverage badge to the README
+* unified some core approaches (e.g., how filters and transformations are applied; tokenizaton)
+* improved test structure
+* improved documentation
+* fixed transformation_usas_en_semtag
+
+## 0.3.3 (2022-09-25)
+
+* added filter_by_filenames
+* added filter_by_filename_not_contains
+* added transform_to_files
+* added transformation_eebop4_to_plaintext
+* added transformation_replace_digits
+* added transformation_ftfy
+* added fast and skip_checkpoint options to load_files
+* added __repr__ method to TextDirectory
+* added examples
+* upgraded to spaCy 3
+* improved the test suite
+* fixed some minor bugs
+
+## 0.3.2 (2021-01-10)
+
+* added transformation_expand_english_contractions
+* fixed some minor bugs
+* added __str__ method to TextDirectory
+* added filename to __str__ output
+* added `get_text` method
+
+## 0.3.1 (2020-01-20)
+
+* added long_description_content_type to setup.py
+
+## 0.3.0 (2020-01-19)
+
+* added transformation_remove_weird_tokens
+* added transformation_lemmatizer
+* fixed some minor bugs
+* added a function to revert applied filters
+* added a function that prints the current pipeline
+* added a function that clears all transformations
+* added helper functions to list available filters and transformations
+* fixed a bug in which `tabulate_flat_list_of_dicts` would fail if the dictionary was empty
+* `self.aggregation` does not hold a copy of the files anymore but references to `self.files`
+* transformations relying on spaCy are now estimating a max_length based on available memory
+* TextDirectory objects are now iterable
+
+## 0.2.2 (2019-06-13)
+
+* changed the data packaging
+
+## 0.2.1 (2019-06-13)
+
+* added transformation_remove_stopwords
+* added transformation_remove_htmltags
+* fixed some minor bugs
+
+## 0.2.0 (2018-05-13)
+
+* added transform_to_memory() function
+* added transformation_to_leetspeak() function
+* added transformation_crude_spellchecker
+* added filter_by_max_filesize
+* added filter_by_min_filesize
+* fixed a bug where load_files() would fail if there were no files
+
+## 0.1.4 (2018-05-02)
+
+* fixed an object mutation problem in the tabulate function
+
+## 0.1.3 (2018-04-30)
+
+* filter_by_random_sampling now has a "replacement" option
+* changed from tabulate to an embedded function
+* added transformation_remove_non_ascii
+* added transformation_remove_non_alphanumerical
+* added filter_by_similar_documents
+
+## 0.1.2 (2018-04-29)
+
+* added transformation_postag
+* added transformation_usas_en_semtag
+* added transformation_uppercase
+* added filter_by_filename_contains
+* added parameter support for transformations
+
+## 0.1.1 (2018-04-27)
+
+* added filter_by_chars_outliers
+* added transformation_remove_nl
+
+## 0.1.0 (2018-04-26)
+
+* Initial release
+* First release on PyPI.

textdirectory-0.4.0/CONTRIBUTING.md

@@ -0,0 +1,53 @@
+# Contributing
+
+Contributions are welcome, and they are greatly appreciated! Every little bit helps, and credit will always be given.
+
+## Ways to Contribute
+
+* **Report bugs** at <https://github.com/IngoKl/textdirectory/issues> — please include your operating system,
+  Python version, and detailed steps to reproduce.
+* **Fix bugs** or **implement features**: anything tagged `bug` or `enhancement` in the issue tracker is open to
+  whoever wants to work on it.
+* **Write documentation**: docstrings, the docs at <https://textdirectory.readthedocs.io>, or blog posts and
+  articles elsewhere.
+* **Submit feedback**: the easiest way is to file an issue at <https://github.com/IngoKl/textdirectory/issues>.
+
+## Development Setup
+
+The project uses [uv](https://docs.astral.sh/uv/) for environments and dependency management.
+
+```bash
+git clone https://github.com/<your-fork>/textdirectory.git
+cd textdirectory
+
+# Create the environment with dev tools and the optional NLP dependencies
+uv sync --group dev --extra nlp
+uv pip install https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.8.0/en_core_web_sm-3.8.0-py3-none-any.whl
+```
+
+Note: if you keep the repository inside a synced folder (e.g. Dropbox), exclude `.venv/` from syncing.
+
+## Running Checks
+
+```bash
+uv run pytest              # test suite (tests hitting live services are excluded by default)
+uv run pytest -m network   # live-service tests (UCREL USAS)
+uv run ruff check src tests
+uv run ruff format --check src tests
+uv run mypy
+```
+
+Tests that need spaCy and the `en_core_web_sm` model are skipped automatically when they are not installed.
+
+## Pull Request Guidelines
+
+1. Create a branch for your change and add tests for any new functionality or fixed bug.
+2. Make sure the full check suite above passes locally.
+3. Update the documentation if you add or change functionality (docstrings, README, and the docs pages).
+4. Add an entry to `CHANGELOG.md` under the unreleased version.
+5. The CI runs the tests on Python 3.10–3.13 on Linux and Windows; your change should pass on all of them.
+
+## Releasing
+
+See the [releasing documentation](https://textdirectory.readthedocs.io/en/latest/releasing.html) for how versions
+are bumped, built, and published to PyPI.

{textdirectory-0.3.2 → textdirectory-0.4.0}/LICENSE

@@ -1,22 +1,22 @@
-MIT License
-
-Copyright (c) 2021, Ingo Kleiber
-
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
-
+MIT License
+
+Copyright (c) 2022, Ingo Kleiber
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+

textdirectory-0.4.0/PKG-INFO

@@ -0,0 +1,262 @@
+Metadata-Version: 2.4
+Name: textdirectory
+Version: 0.4.0
+Summary: Aggregate multiple text files into one, with filters and transformations.
+Project-URL: Homepage, https://github.com/IngoKl/textdirectory
+Project-URL: Documentation, https://textdirectory.readthedocs.io
+Project-URL: Repository, https://github.com/IngoKl/textdirectory
+Project-URL: Issues, https://github.com/IngoKl/textdirectory/issues
+Author-email: Ingo Kleiber <ingo@kleiber.me>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: aggregation,corpus,nlp,text,textdirectory
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Natural Language :: English
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Requires-Python: >=3.10
+Requires-Dist: beautifulsoup4>=4.11
+Requires-Dist: click>=8.1
+Requires-Dist: ftfy>=6.1
+Requires-Dist: lxml>=4.9
+Requires-Dist: requests>=2.28
+Requires-Dist: tqdm>=4.64
+Provides-Extra: nlp
+Requires-Dist: psutil>=5.9; extra == 'nlp'
+Requires-Dist: spacy<4,>=3.8; extra == 'nlp'
+Description-Content-Type: text/markdown
+
+# TextDirectory
+
+[![PyPI](https://img.shields.io/pypi/v/textdirectory.svg)](https://pypi.org/project/textdirectory/)
+[![CI](https://github.com/IngoKl/textdirectory/actions/workflows/ci.yml/badge.svg)](https://github.com/IngoKl/textdirectory/actions/workflows/ci.yml)
+[![Documentation Status](https://readthedocs.org/projects/textdirectory/badge/?version=latest)](https://textdirectory.readthedocs.io/en/latest/?badge=latest)
+[![Python Versions](https://img.shields.io/pypi/pyversions/textdirectory.svg)](https://pypi.org/project/textdirectory/)
+
+![TextDirectory](https://user-images.githubusercontent.com/16179317/39367680-cd409a00-4a37-11e8-8d42-0bed5a4e814b.png)
+
+*TextDirectory* allows you to combine multiple text files into one aggregated file. TextDirectory also supports matching
+files for certain criteria and applying transformations to the aggregated text.
+
+*TextDirectory* can be used as a mere tool (via the CLI) and as a Python library.
+
+Of course, everything *TextDirectory* does could be achieved in bash or PowerShell. However, there are certain
+use cases (e.g., when used as a library) in which it might be useful.
+
+* Free software: MIT license
+* Documentation: <https://textdirectory.readthedocs.io>
+
+## Features
+
+* Aggregating multiple text files
+* Filtering documents/texts based on various parameters such as length, content, and random sampling
+* Filtering and transforming text files
+* Transforming the aggregated text (e.g., transforming the text to lowercase)
+
+| Version | Filters | Transformations |
+|---------|---------|-----------------|
+| 0.1.0 | filter_by_max_chars(n int); filter_by_min_chars(n int); filter_by_max_tokens(n int); filter_by_min_tokens(n int); filter_by_contains(str); filter_by_not_contains(str); filter_by_random_sampling(n int; replace=False) | transformation_lowercase |
+| 0.1.1 | filter_by_chars_outliers(n sigmas int) | transformation_remove_nl |
+| 0.1.2 | filter_by_filename_contains(contains str) | transformation_usas_en_semtag; transformation_uppercase; transformation_postag(spacy_model str) |
+| 0.1.3 | filter_by_similar_documents(reference_file str; threshold float) | transformation_remove_non_ascii; transformation_remove_non_alphanumerical |
+| 0.2.0 | filter_by_max_filesize(max_kb int); filter_by_min_filesize(min_kb int) | transformation_to_leetspeak; transformation_crude_spellchecker(language model str) |
+| 0.2.1 | None | transformation_remove_stopwords(stopwords_source str; stopwords str [en]; spacy_model str; custom_stopwords str); transformation_remove_htmltags |
+| 0.3.0 | None | transformation_remove_weird_tokens(spaCy model; remove_double_space=False); transformation_lemmatize(spaCy model) |
+| 0.3.2 | None | transformation_expand_english_contractions |
+| 0.3.3 | filter_by_filenames(filenames list); filter_by_filename_not_contains(not_contains str) | transformation_eebop4_to_plaintext; transformation_replace_digits(replacement_character str); transformation_ftfy |
+| 0.3.4 | filter_by_type_token_ratio(min_ttr float; max_ttr float) | transformation_replace_string(replace str; replace_with str) |
+| 0.4.0 | None (modernization release) | None (modernization release) |
+
+## Installation
+
+Install *TextDirectory* via pip:
+
+```bash
+pip install textdirectory
+```
+
+The spaCy-based transformations (`transformation_postag`, `transformation_lemmatize`,
+`transformation_remove_stopwords`, `transformation_remove_weird_tokens`) are optional. To use them, install the
+`nlp` extra and download a spaCy model:
+
+```bash
+pip install 'textdirectory[nlp]'
+python -m spacy download en_core_web_sm
+```
+
+## Quickstart
+
+*TextDirectory*, as exemplified below, works with a two-stage model. After loading in your data (directory) you can,
+by applying *filters*, iteratively select the files you want to process.
+In a second step you can perform transformations on the text before finally aggregating it to either a file or memory.
+
+![TextDirectory](https://user-images.githubusercontent.com/16179317/39367589-7f774116-4a37-11e8-9a09-5cbdf5f3311b.png)
+
+### As a Command-Line Tool
+
+*TextDirectory* comes equipped with a CLI (`textdirectory`, also available as `python -m textdirectory`).
+
+The syntax for both the *filters* and *transformations* works similarly. They are chained by adding slashes (/) and
+parameters are passed via commas (,): `filter_by_min_tokens,5/filter_by_random_sampling,2`.
+
+**Example 1: A Very Simple Aggregation**
+
+```bash
+textdirectory --directory testdata --output_file aggregated.txt
+```
+
+This will take all files (.txt) in *testdata* and then aggregates the files into a file called *aggregated.txt*.
+
+You could also use '*' as a wildcard for filetype if you need to include all files and not just .txt:
+
+```bash
+textdirectory --directory testdata --output_file aggregated.txt --filetype *
+```
+
+**Example 2: Applying Filters and Transformations**
+
+In this example, we want to filter the files based on their token count, perform a random sampling and finally
+transform all text to lowercase:
+
+```bash
+textdirectory --directory testdata --output_file aggregated.txt --filters filter_by_min_tokens,5/filter_by_random_sampling,2 --transformations transformation_lowercase
+```
+
+After passing two filters (*filter_by_min_tokens* and *filter_by_random_sampling*) we've applied the
+*transformation_lowercase* transformation.
+
+The resulting file will contain the content of two files that each have at least five tokens.
+
+The same transformation can also be staged multiple times. For example, we could stage two simple string replacements
+like this:
+
+```bash
+textdirectory --directory testdata --output_file aggregated.txt --transformations transformation_replace_string,lorem,x/transformation_replace_string,ipsum,y
+```
+
+If the `--output_file` is missing, the CLI will output the aggregated text to the console. In this scenario, there is
+no progress bar visible (can be controlled using `--disable_tqdm True/False`).
+
+### As a Python Library
+
+In order to demonstrate *TextDirectory* as a Python library, we'll recreate the second example from above:
+
+```python
+import textdirectory
+
+td = textdirectory.TextDirectory(directory='testdata')
+td.load_files(recursive=False, filetype='txt', sort=True)
+td.filter_by_min_tokens(5)
+td.filter_by_random_sampling(2)
+td.stage_transformation(['transformation_lowercase'])
+td.aggregate_to_file('aggregated.txt')
+```
+
+If we don't have special requirements, we can also call
+`td = textdirectory.TextDirectory(directory='testdata', autoload=True)` to skip manually calling `load_files`.
+If we wanted to keep working with the actual aggregated text, we could have called `text = td.aggregate_to_memory()`
+instead of `aggregate_to_file`.
+
+```python
+import textdirectory
+
+td = textdirectory.TextDirectory(directory='testdata', autoload=True)
+td.get_text(0)
+```
+
+Sometimes we might want to get the actual text of a given file. This can be achieved as seen above. The `get_text`
+method will return the transformed text if it is available. Otherwise, it will simply read the file and return the
+text.
+
+Every applied filter will create a *state* (i.e., a checkpoint). If we want to go back to a previous state, we can
+print all states by calling `td.print_saved_states()`. Previous states can then be loaded by calling
+`td.load_aggregation_state(state=0)`.
+
+It's also possible to pass arguments to the individual transformations. In order to do this (at the moment) you have
+to adhere to the correct order of arguments:
+
+```python
+# def transformation_remove_stopwords(text, stopwords_source='internal', stopwords='en', spacy_model='en_core_web_sm', custom_stopwords=None, *args)
+td.stage_transformation(['transformation_remove_stopwords', 'internal', 'en', 'en_core_web_sm', 'dolor'])
+```
+
+In the above example, we are adding additional custom stopwords to the transformer.
+
+You also might not always want to aggregate texts into one file in many cases but filter and transform them:
+
+```python
+import textdirectory
+
+td = textdirectory.TextDirectory(directory='input')
+td.load_files()
+
+td.filter_by_max_chars(480)
+td.stage_transformation(['transformation_to_leetspeak'])
+
+td.transform_to_files('output')
+```
+
+In the example above, we are loading all files in `input`. After filtering and transforming, the modified files will
+be written to `output`.
+
+In addition, there are a few [simple examples](https://github.com/IngoKl/textdirectory/tree/master/examples) in the
+repository for you to look at.
+
+## Special Transformations
+
+### transformation_crude_spellchecker
+
+This is a very crude spellchecker implemented following Peter Norvig's excellent
+[tutorial](http://norvig.com/spell-correct.html). It's really more of an experiment and not production ready in any
+sense of the word.
+
+### transformation_eebop4_to_plaintext
+
+This is a highly specific transformation that will extract the plain text from an EEBO-TCP P4 corpus file.
+Both the header as well as all XML tags will be removed during this transformation.
+
+### transformation_ftfy
+
+This transformation simply applies `ftfy.fix_text` to the text.
+It is highly recommended to use [ftfy](https://pypi.org/project/ftfy/) as a first transformation if you are working
+with messy Unicode text.
+
+## Notes for Developers
+
+The project uses [uv](https://docs.astral.sh/uv/) for development:
+
+```bash
+uv sync --group dev --extra nlp
+uv pip install https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.8.0/en_core_web_sm-3.8.0-py3-none-any.whl
+
+uv run pytest              # run the test suite (network tests excluded by default)
+uv run pytest -m network   # run the tests that hit live external services
+uv run ruff check src tests
+uv run mypy
+```
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) for the full development guide and
+[the releasing documentation](https://textdirectory.readthedocs.io/en/latest/releasing.html) for how releases are
+built and published.
+
+## Behavior
+
+We are not holding the actual texts in memory. This leads to much more disk read activity (and time inefficiency),
+but saves memory. Of course, this is not the case when using `aggregate_to_memory`.
+
+`transformation_usas_en_semtag` relies on the web version of
+[Paul Rayson's USAS Tagger](http://ucrel.lancs.ac.uk/usas/). Don't use this transformation for large amounts of text,
+give credit, and consider using their commercial product [Wmatrix](http://ucrel.lancs.ac.uk/wmatrix/).
+
+If you are working with a lot of files, it might be wise to use `load_files(fast=True, skip_checkpoint=True)`.
+This will load files much quicker but skip collecting metadata. This will limit the filters that you can use.
+
+## Credits
+
+This package was originally based on the
+[audreyr/cookiecutter-pypackage](https://github.com/audreyr/cookiecutter-pypackage) cookiecutter template.