nlp-text-preprocessing 0.5.0__tar.gz → 0.5.1__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (32) hide show
  1. {nlp_text_preprocessing-0.5.0 → nlp_text_preprocessing-0.5.1}/PKG-INFO +100 -2
  2. nlp_text_preprocessing-0.5.0/nlp_text_preprocessing.egg-info/PKG-INFO → nlp_text_preprocessing-0.5.1/Readme.md +536 -499
  3. {nlp_text_preprocessing-0.5.0 → nlp_text_preprocessing-0.5.1}/nlp_text_preprocessing/__init__.py +1 -1
  4. {nlp_text_preprocessing-0.5.0 → nlp_text_preprocessing-0.5.1}/nlp_text_preprocessing/all_text_preprocess.py +6 -4
  5. nlp_text_preprocessing-0.5.0/Readme.md → nlp_text_preprocessing-0.5.1/nlp_text_preprocessing.egg-info/PKG-INFO +597 -438
  6. {nlp_text_preprocessing-0.5.0 → nlp_text_preprocessing-0.5.1}/tests/test_infrastructure.py +1 -1
  7. {nlp_text_preprocessing-0.5.0 → nlp_text_preprocessing-0.5.1}/tests/test_new_features.py +9 -0
  8. {nlp_text_preprocessing-0.5.0 → nlp_text_preprocessing-0.5.1}/LICENSE +0 -0
  9. {nlp_text_preprocessing-0.5.0 → nlp_text_preprocessing-0.5.1}/MANIFEST.in +0 -0
  10. {nlp_text_preprocessing-0.5.0 → nlp_text_preprocessing-0.5.1}/architecture_diagram.png +0 -0
  11. {nlp_text_preprocessing-0.5.0 → nlp_text_preprocessing-0.5.1}/nlp_banner.png +0 -0
  12. {nlp_text_preprocessing-0.5.0 → nlp_text_preprocessing-0.5.1}/nlp_text_preprocessing/__main__.py +0 -0
  13. {nlp_text_preprocessing-0.5.0 → nlp_text_preprocessing-0.5.1}/nlp_text_preprocessing/cli.py +0 -0
  14. {nlp_text_preprocessing-0.5.0 → nlp_text_preprocessing-0.5.1}/nlp_text_preprocessing/data/contractions.json +0 -0
  15. {nlp_text_preprocessing-0.5.0 → nlp_text_preprocessing-0.5.1}/nlp_text_preprocessing/py.typed +0 -0
  16. {nlp_text_preprocessing-0.5.0 → nlp_text_preprocessing-0.5.1}/nlp_text_preprocessing/sklearn_compat.py +0 -0
  17. {nlp_text_preprocessing-0.5.0 → nlp_text_preprocessing-0.5.1}/nlp_text_preprocessing.egg-info/SOURCES.txt +0 -0
  18. {nlp_text_preprocessing-0.5.0 → nlp_text_preprocessing-0.5.1}/nlp_text_preprocessing.egg-info/dependency_links.txt +0 -0
  19. {nlp_text_preprocessing-0.5.0 → nlp_text_preprocessing-0.5.1}/nlp_text_preprocessing.egg-info/entry_points.txt +0 -0
  20. {nlp_text_preprocessing-0.5.0 → nlp_text_preprocessing-0.5.1}/nlp_text_preprocessing.egg-info/requires.txt +0 -0
  21. {nlp_text_preprocessing-0.5.0 → nlp_text_preprocessing-0.5.1}/nlp_text_preprocessing.egg-info/top_level.txt +0 -0
  22. {nlp_text_preprocessing-0.5.0 → nlp_text_preprocessing-0.5.1}/pyproject.toml +0 -0
  23. {nlp_text_preprocessing-0.5.0 → nlp_text_preprocessing-0.5.1}/requirements.txt +0 -0
  24. {nlp_text_preprocessing-0.5.0 → nlp_text_preprocessing-0.5.1}/setup.cfg +0 -0
  25. {nlp_text_preprocessing-0.5.0 → nlp_text_preprocessing-0.5.1}/tests/test_all_text_preprocess.py +0 -0
  26. {nlp_text_preprocessing-0.5.0 → nlp_text_preprocessing-0.5.1}/tests/test_batching.py +0 -0
  27. {nlp_text_preprocessing-0.5.0 → nlp_text_preprocessing-0.5.1}/tests/test_cleaning.py +0 -0
  28. {nlp_text_preprocessing-0.5.0 → nlp_text_preprocessing-0.5.1}/tests/test_known_limits.py +0 -0
  29. {nlp_text_preprocessing-0.5.0 → nlp_text_preprocessing-0.5.1}/tests/test_light_and_cli.py +0 -0
  30. {nlp_text_preprocessing-0.5.0 → nlp_text_preprocessing-0.5.1}/tests/test_linguistic.py +0 -0
  31. {nlp_text_preprocessing-0.5.0 → nlp_text_preprocessing-0.5.1}/tests/test_moderation.py +0 -0
  32. {nlp_text_preprocessing-0.5.0 → nlp_text_preprocessing-0.5.1}/tests/test_pipeline_api.py +0 -0
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: nlp-text-preprocessing
3
- Version: 0.5.0
3
+ Version: 0.5.1
4
4
  Summary: Text cleaning, feature extraction and linguistic utilities for NLP and ML pipelines
5
5
  Author-email: Uditya Narayan Tiwari <tiwarimerit@gmail.com>
6
6
  License-Expression: MIT
@@ -88,6 +88,7 @@ Dynamic: license-file
88
88
  ## Table of Contents
89
89
 
90
90
  - [Package Overview](#package-overview)
91
+ - [What's New in 0.5.0](#whats-new-in-050)
91
92
  - [Key Features](#key-features)
92
93
  - [Architecture Diagram](#architecture-diagram)
93
94
  - [Quickstart and Setup](#quickstart-and-setup)
@@ -106,6 +107,11 @@ Dynamic: license-file
106
107
  - [Pandas DataFrame Integration](#3-pandas-dataframe-integration)
107
108
  - [Batch Feature Extraction](#4-batch-feature-extraction)
108
109
  - [WordCloud Generation](#5-wordcloud-generation)
110
+ - [Readability Scores](#6-readability-scores)
111
+ - [Masking Personal Data, Including Names](#7-masking-personal-data-including-names)
112
+ - [Fast Batch Processing](#8-fast-batch-processing)
113
+ - [Type Hints](#type-hints)
114
+ - [Development](#development)
109
115
  - [Troubleshooting and Resource Setup](#troubleshooting-and-resource-setup)
110
116
  - [License and Author](#license-and-author)
111
117
 
@@ -121,16 +127,30 @@ It provides over **60 single-purpose functions** and an extensible **Pipeline En
121
127
 
122
128
  ---
123
129
 
130
+ ## What's New in 0.5.0
131
+
132
+ - **`readability_scores(x)`**: five readability measures in one dictionary (Flesch Reading Ease, Flesch-Kincaid grade, Gunning Fog, ARI, Coleman-Liau). See [Readability Scores](#6-readability-scores).
133
+ - **`mask_pii(x, names=True)`**: also redacts people's names with spaCy's entity recogniser. Off by default. See [Masking Personal Data](#7-masking-personal-data-including-names).
134
+ - **Faster batches**: `Pipeline.run_batch`, `transform` and `run_series` process the spaCy steps with `nlp.pipe` (about 2x faster, identical output).
135
+ - **Type hints** on the public API and a `py.typed` marker, so mypy, Pyright and IDEs use them. See [Type Hints](#type-hints).
136
+ - **Developer tooling**: ruff, mypy, coverage, Python 3.13 in CI and Dependabot. See [Development](#development).
137
+
138
+ Everything from earlier versions still works the same way. The full history is in [CHANGELOG.md](CHANGELOG.md).
139
+
140
+ ---
141
+
124
142
  ## Key Features
125
143
 
126
144
  - **Safe Inputs**: `None`, `NaN`, `pd.NA` and bytes are handled by every function (treated as empty text).
127
145
  - **Pipeline Automation Engine**: Chain cleaning steps by name or custom functions via `Pipeline(['to_lower_case', 'remove_urls', 'lemmatize'])`.
128
146
  - **Light Import, Lazy Models**: `import nlp_text_preprocessing` loads no heavy library and downloads nothing. spaCy, TextBlob and NLTK data are loaded on first use and cached.
129
147
  - **Advanced Cleaning**: Remove emojis, phone numbers, PII mask, URLs, emails, HTML, retweets, special symbols, and expand contractions/hashtags.
130
- - **Feature & Readability Extraction**: Extract word counts, sentence counts, readability scores (Flesch Ease), punctuation density, capital ratios, and full batch statistics.
148
+ - **Feature & Readability Extraction**: Extract word counts, sentence counts, readability scores (Flesch Ease, plus Flesch-Kincaid, Gunning Fog, ARI and Coleman-Liau via `readability_scores`), punctuation density, capital ratios, and full batch statistics.
131
149
  - **Linguistic Utilities**: Named Entity Recognition (NER), POS tagging, dependency parsing, language detection, keyword extraction, synonyms, antonyms, and n-grams.
132
150
  - **Safety & Moderation (baseline)**: Lexicon-based profanity, toxicity, spam and emotion helpers. They are transparent keyword baselines, not a moderation system: use a trained model for anything high-stakes.
133
151
  - **Visualization**: Generate and save custom WordClouds to disk or display interactively.
152
+ - **Fast Batches**: `Pipeline.run_batch` / `run_series` use spaCy's `nlp.pipe` for the spaCy steps.
153
+ - **Typed**: annotated public API with a `py.typed` marker.
134
154
 
135
155
  ---
136
156
 
@@ -153,7 +173,9 @@ pip install nlp-text-preprocessing # core
153
173
  pip install "nlp-text-preprocessing[viz]" # + wordcloud, matplotlib (get_wordcloud)
154
174
  pip install "nlp-text-preprocessing[pandas]" # + pandas (text_stats_batch, Pipeline.run_series)
155
175
  pip install "nlp-text-preprocessing[sklearn]" # + scikit-learn (TextPreprocessor)
176
+ pip install "nlp-text-preprocessing[transformers]" # + transformers (tokenizers for truncate_text / chunk_text)
156
177
  pip install "nlp-text-preprocessing[all]" # everything optional
178
+ pip install "nlp-text-preprocessing[dev]" # tests, ruff, mypy (for contributors)
157
179
  ```
158
180
 
159
181
  ### 2. Downloading Required Corpora and Models
@@ -474,6 +496,82 @@ text = "python natural language processing machine learning text mining data sci
474
496
  tp.get_wordcloud(text, save_path="wordcloud.png", show=False)
475
497
  ```
476
498
 
499
+ ### 6. Readability Scores
500
+
501
+ ```python
502
+ import nlp_text_preprocessing as tp
503
+
504
+ scores = tp.readability_scores("The cat sat on the mat. It was a sunny day.")
505
+ print(scores)
506
+ # {'flesch_reading_ease': ..., 'flesch_kincaid_grade': ..., 'gunning_fog': ...,
507
+ # 'automated_readability_index': ..., 'coleman_liau': ...}
508
+
509
+ # Higher Flesch = easier. The other four are US school grade levels (higher = harder).
510
+ # Empty or None text returns 0.0 for every key.
511
+ ```
512
+
513
+ `tp.readability_score(x)` still returns only the Flesch Reading Ease number. The scores are approximate and meant for English.
514
+
515
+ ### 7. Masking Personal Data, Including Names
516
+
517
+ ```python
518
+ import nlp_text_preprocessing as tp
519
+
520
+ text = "Barack Obama emailed me at obama@example.com or 555-123-4567."
521
+
522
+ tp.mask_pii(text)
523
+ # 'Barack Obama emailed me at [REDACTED] or [REDACTED].' (names are left alone by default)
524
+
525
+ tp.mask_pii(text, names=True)
526
+ # '[REDACTED] emailed me at [REDACTED] or [REDACTED].' (needs the spaCy model; run tp.download_nltk_packages() once)
527
+
528
+ tp.mask_pii(text, token='<PII>', names=True) # custom replacement token
529
+ ```
530
+
531
+ Names are found by spaCy's statistical entity recogniser, so it can miss or over-match some names. Review the output when privacy matters.
532
+
533
+ ### 8. Fast Batch Processing
534
+
535
+ ```python
536
+ import nlp_text_preprocessing as tp
537
+
538
+ pipe = tp.Pipeline(['to_lower_case', 'remove_urls', 'lemmatize'])
539
+
540
+ texts = ["Dogs are barking at https://example.com", "Cats were running"] * 500
541
+ clean = pipe.run_batch(texts) # list in, list out
542
+ clean_series = pipe.run_series(df['text']) # keeps the Series index and name
543
+ ```
544
+
545
+ ---
546
+
547
+ ## Type Hints
548
+
549
+ The package ships a `py.typed` marker and annotated function signatures, so type checkers and IDEs show parameter and return types:
550
+
551
+ ```python
552
+ import nlp_text_preprocessing as tp
553
+
554
+ n: int = tp.word_count("hello world")
555
+ scores: dict = tp.readability_scores("Hello there.")
556
+ ```
557
+
558
+ ---
559
+
560
+ ## Development
561
+
562
+ ```bash
563
+ git clone https://github.com/udityamerit/Text-Processing-Package-For-Natural-Language-Processing.git
564
+ cd Text-Processing-Package-For-Natural-Language-Processing
565
+ pip install -e ".[dev,all]"
566
+ python -m spacy download en_core_web_sm
567
+
568
+ pytest -q --cov # tests with coverage
569
+ ruff check . # lint
570
+ mypy # type check
571
+ ```
572
+
573
+ CI runs the same checks on Python 3.10 to 3.13. Release steps, and how to set up PyPI Trusted Publishing, are in [RELEASING.md](RELEASING.md).
574
+
477
575
  ---
478
576
 
479
577
  ## Troubleshooting and Resource Setup