nlp-text-preprocessing 0.3.0__tar.gz → 0.4.1__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {nlp_text_preprocessing-0.3.0 → nlp_text_preprocessing-0.4.1}/PKG-INFO +99 -7
- nlp_text_preprocessing-0.3.0/nlp_text_preprocessing.egg-info/PKG-INFO → nlp_text_preprocessing-0.4.1/Readme.md +435 -401
- {nlp_text_preprocessing-0.3.0 → nlp_text_preprocessing-0.4.1}/nlp_text_preprocessing/__init__.py +14 -1
- nlp_text_preprocessing-0.4.1/nlp_text_preprocessing/__main__.py +5 -0
- {nlp_text_preprocessing-0.3.0 → nlp_text_preprocessing-0.4.1}/nlp_text_preprocessing/all_text_preprocess.py +463 -51
- nlp_text_preprocessing-0.4.1/nlp_text_preprocessing/cli.py +150 -0
- nlp_text_preprocessing-0.4.1/nlp_text_preprocessing/sklearn_compat.py +44 -0
- nlp_text_preprocessing-0.3.0/Readme.md → nlp_text_preprocessing-0.4.1/nlp_text_preprocessing.egg-info/PKG-INFO +493 -349
- {nlp_text_preprocessing-0.3.0 → nlp_text_preprocessing-0.4.1}/nlp_text_preprocessing.egg-info/SOURCES.txt +9 -1
- nlp_text_preprocessing-0.4.1/nlp_text_preprocessing.egg-info/entry_points.txt +2 -0
- {nlp_text_preprocessing-0.3.0 → nlp_text_preprocessing-0.4.1}/nlp_text_preprocessing.egg-info/requires.txt +8 -0
- {nlp_text_preprocessing-0.3.0 → nlp_text_preprocessing-0.4.1}/pyproject.toml +7 -2
- nlp_text_preprocessing-0.4.1/requirements.txt +15 -0
- {nlp_text_preprocessing-0.3.0 → nlp_text_preprocessing-0.4.1}/tests/test_infrastructure.py +1 -1
- nlp_text_preprocessing-0.4.1/tests/test_known_limits.py +38 -0
- nlp_text_preprocessing-0.4.1/tests/test_light_and_cli.py +195 -0
- nlp_text_preprocessing-0.4.1/tests/test_moderation.py +102 -0
- nlp_text_preprocessing-0.4.1/tests/test_pipeline_api.py +100 -0
- nlp_text_preprocessing-0.3.0/requirements.txt +0 -7
- {nlp_text_preprocessing-0.3.0 → nlp_text_preprocessing-0.4.1}/LICENSE +0 -0
- {nlp_text_preprocessing-0.3.0 → nlp_text_preprocessing-0.4.1}/MANIFEST.in +0 -0
- {nlp_text_preprocessing-0.3.0 → nlp_text_preprocessing-0.4.1}/architecture_diagram.png +0 -0
- {nlp_text_preprocessing-0.3.0 → nlp_text_preprocessing-0.4.1}/nlp_banner.png +0 -0
- {nlp_text_preprocessing-0.3.0 → nlp_text_preprocessing-0.4.1}/nlp_text_preprocessing/data/contractions.json +0 -0
- {nlp_text_preprocessing-0.3.0 → nlp_text_preprocessing-0.4.1}/nlp_text_preprocessing.egg-info/dependency_links.txt +0 -0
- {nlp_text_preprocessing-0.3.0 → nlp_text_preprocessing-0.4.1}/nlp_text_preprocessing.egg-info/top_level.txt +0 -0
- {nlp_text_preprocessing-0.3.0 → nlp_text_preprocessing-0.4.1}/setup.cfg +0 -0
- {nlp_text_preprocessing-0.3.0 → nlp_text_preprocessing-0.4.1}/tests/test_all_text_preprocess.py +0 -0
- {nlp_text_preprocessing-0.3.0 → nlp_text_preprocessing-0.4.1}/tests/test_cleaning.py +0 -0
- {nlp_text_preprocessing-0.3.0 → nlp_text_preprocessing-0.4.1}/tests/test_linguistic.py +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: nlp-text-preprocessing
|
|
3
|
-
Version: 0.
|
|
3
|
+
Version: 0.4.1
|
|
4
4
|
Summary: Text cleaning, feature extraction and linguistic utilities for NLP and ML pipelines
|
|
5
5
|
Author-email: Uditya Narayan Tiwari <tiwarimerit@gmail.com>
|
|
6
6
|
License-Expression: MIT
|
|
@@ -37,10 +37,15 @@ Requires-Dist: pandas; extra == "pandas"
|
|
|
37
37
|
Provides-Extra: viz
|
|
38
38
|
Requires-Dist: wordcloud; extra == "viz"
|
|
39
39
|
Requires-Dist: matplotlib; extra == "viz"
|
|
40
|
+
Provides-Extra: sklearn
|
|
41
|
+
Requires-Dist: scikit-learn; extra == "sklearn"
|
|
42
|
+
Provides-Extra: transformers
|
|
43
|
+
Requires-Dist: transformers; extra == "transformers"
|
|
40
44
|
Provides-Extra: all
|
|
41
45
|
Requires-Dist: pandas; extra == "all"
|
|
42
46
|
Requires-Dist: wordcloud; extra == "all"
|
|
43
47
|
Requires-Dist: matplotlib; extra == "all"
|
|
48
|
+
Requires-Dist: scikit-learn; extra == "all"
|
|
44
49
|
Provides-Extra: dev
|
|
45
50
|
Requires-Dist: pytest; extra == "dev"
|
|
46
51
|
Requires-Dist: build; extra == "dev"
|
|
@@ -48,6 +53,7 @@ Requires-Dist: twine; extra == "dev"
|
|
|
48
53
|
Requires-Dist: pandas; extra == "dev"
|
|
49
54
|
Requires-Dist: wordcloud; extra == "dev"
|
|
50
55
|
Requires-Dist: matplotlib; extra == "dev"
|
|
56
|
+
Requires-Dist: scikit-learn; extra == "dev"
|
|
51
57
|
Dynamic: license-file
|
|
52
58
|
|
|
53
59
|
# NLP Text Preprocessing Utility Package
|
|
@@ -83,6 +89,8 @@ Dynamic: license-file
|
|
|
83
89
|
- [Architecture Diagram](#architecture-diagram)
|
|
84
90
|
- [Quickstart and Setup](#quickstart-and-setup)
|
|
85
91
|
- [Pipeline Automation Engine](#pipeline-automation-engine)
|
|
92
|
+
- [Preparing Text for Transformers](#preparing-text-for-transformers-bert-gpt-)
|
|
93
|
+
- [Command Line (`nlp-prep`)](#command-line-nlp-prep)
|
|
86
94
|
- [Complete API Reference](#complete-api-reference)
|
|
87
95
|
- [1. General Feature Extraction](#1-general-feature-extraction)
|
|
88
96
|
- [2. Text Cleaning and Normalization](#2-text-cleaning-and-normalization)
|
|
@@ -118,7 +126,7 @@ It provides over **60 single-purpose functions** and an extensible **Pipeline En
|
|
|
118
126
|
- **Advanced Cleaning**: Remove emojis, phone numbers, PII mask, URLs, emails, HTML, retweets, special symbols, and expand contractions/hashtags.
|
|
119
127
|
- **Feature & Readability Extraction**: Extract word counts, sentence counts, readability scores (Flesch Ease), punctuation density, capital ratios, and full batch statistics.
|
|
120
128
|
- **Linguistic Utilities**: Named Entity Recognition (NER), POS tagging, dependency parsing, language detection, keyword extraction, synonyms, antonyms, and n-grams.
|
|
121
|
-
- **Safety & Moderation**:
|
|
129
|
+
- **Safety & Moderation (baseline)**: Lexicon-based profanity, toxicity, spam and emotion helpers. They are transparent keyword baselines, not a moderation system: use a trained model for anything high-stakes.
|
|
122
130
|
- **Visualization**: Generate and save custom WordClouds to disk or display interactively.
|
|
123
131
|
|
|
124
132
|
---
|
|
@@ -141,6 +149,7 @@ The package integrates **spaCy**, **NLTK**, **TextBlob**, **BeautifulSoup4**, an
|
|
|
141
149
|
pip install nlp-text-preprocessing # core
|
|
142
150
|
pip install "nlp-text-preprocessing[viz]" # + wordcloud, matplotlib (get_wordcloud)
|
|
143
151
|
pip install "nlp-text-preprocessing[pandas]" # + pandas (text_stats_batch, Pipeline.run_series)
|
|
152
|
+
pip install "nlp-text-preprocessing[sklearn]" # + scikit-learn (TextPreprocessor)
|
|
144
153
|
pip install "nlp-text-preprocessing[all]" # everything optional
|
|
145
154
|
```
|
|
146
155
|
|
|
@@ -192,6 +201,85 @@ clean_series = pipe.run_series(df['text_column'])
|
|
|
192
201
|
report = tp.diff_report(raw_texts, clean_batch)
|
|
193
202
|
```
|
|
194
203
|
|
|
204
|
+
### Steps with arguments, saving and scikit-learn
|
|
205
|
+
|
|
206
|
+
```python
|
|
207
|
+
# Pass arguments as (name, {argument: value})
|
|
208
|
+
pipe = tp.Pipeline([
|
|
209
|
+
'to_lower_case',
|
|
210
|
+
('remove_stop_words', {'keep_negations': False}),
|
|
211
|
+
])
|
|
212
|
+
print(pipe) # Pipeline(['to_lower_case', "remove_stop_words{'keep_negations': False}"])
|
|
213
|
+
|
|
214
|
+
# Save and reload (custom functions must be registered with @tp.register_step('name'))
|
|
215
|
+
pipe.save('my_pipeline.json')
|
|
216
|
+
pipe = tp.Pipeline.load('my_pipeline.json')
|
|
217
|
+
|
|
218
|
+
# Use it inside scikit-learn: pip install "nlp-text-preprocessing[sklearn]"
|
|
219
|
+
from sklearn.pipeline import make_pipeline
|
|
220
|
+
from sklearn.feature_extraction.text import TfidfVectorizer
|
|
221
|
+
|
|
222
|
+
model = make_pipeline(tp.TextPreprocessor(['to_lower_case', 'remove_special_chars']),
|
|
223
|
+
TfidfVectorizer())
|
|
224
|
+
X = model.fit_transform(["Hello, World!", "HELLO there"])
|
|
225
|
+
```
|
|
226
|
+
|
|
227
|
+
Every step except the last must return a string. The last step may return anything,
|
|
228
|
+
for example `Pipeline(['to_lower_case', 'text_stats'])`.
|
|
229
|
+
|
|
230
|
+
---
|
|
231
|
+
|
|
232
|
+
## Preparing Text for Transformers (BERT, GPT, ...)
|
|
233
|
+
|
|
234
|
+
Modern models work best with *lightly* cleaned text: keep the case, punctuation and
|
|
235
|
+
word forms, and only remove noise. `light_clean` does exactly that, and `truncate_text`
|
|
236
|
+
/ `chunk_text` help you stay inside a model's token limit.
|
|
237
|
+
|
|
238
|
+
```python
|
|
239
|
+
import nlp_text_preprocessing as tp
|
|
240
|
+
|
|
241
|
+
tp.light_clean("<p>Great & fast! Visit https://x.com or mail me@x.com</p>",
|
|
242
|
+
urls='[URL]', emails='[EMAIL]')
|
|
243
|
+
# 'Great & fast! Visit [URL] or mail [EMAIL]'
|
|
244
|
+
|
|
245
|
+
# Use the real tokenizer so the limits are counted in model tokens
|
|
246
|
+
# (pip install "nlp-text-preprocessing[transformers]")
|
|
247
|
+
from transformers import AutoTokenizer
|
|
248
|
+
tok = AutoTokenizer.from_pretrained('bert-base-uncased')
|
|
249
|
+
|
|
250
|
+
tp.truncate_text(long_text, 256, tokenizer=tok)
|
|
251
|
+
chunks = tp.chunk_text(long_text, 256, overlap=32, tokenizer=tok)
|
|
252
|
+
```
|
|
253
|
+
|
|
254
|
+
Without a `tokenizer`, the limits are counted in whitespace-separated words.
|
|
255
|
+
`tokenizer` may be a HuggingFace tokenizer (anything with `.tokenize()` and
|
|
256
|
+
`.convert_tokens_to_string()`) or a plain function that returns a list of tokens.
|
|
257
|
+
|
|
258
|
+
---
|
|
259
|
+
|
|
260
|
+
## Command Line (`nlp-prep`)
|
|
261
|
+
|
|
262
|
+
Installing the package also installs the `nlp-prep` command (`python -m nlp_text_preprocessing` works too).
|
|
263
|
+
|
|
264
|
+
```bash
|
|
265
|
+
# one document per line, written to a new file
|
|
266
|
+
nlp-prep clean reviews.txt -o reviews_clean.txt
|
|
267
|
+
|
|
268
|
+
# a CSV column (adds a 'text_clean' column; use --output-column text to overwrite)
|
|
269
|
+
nlp-prep clean data.csv --column text -o data_clean.csv
|
|
270
|
+
|
|
271
|
+
# standard input / output
|
|
272
|
+
cat notes.txt | nlp-prep clean - --lowercase --url-token "[URL]"
|
|
273
|
+
|
|
274
|
+
# classic cleaning, or your own steps / a saved Pipeline
|
|
275
|
+
nlp-prep clean reviews.txt --mode standard --stopwords
|
|
276
|
+
nlp-prep clean reviews.txt --steps to_lower_case,remove_urls
|
|
277
|
+
nlp-prep clean reviews.txt --pipeline my_pipeline.json
|
|
278
|
+
|
|
279
|
+
# check or install the NLTK data and spaCy model
|
|
280
|
+
nlp-prep resources --download
|
|
281
|
+
```
|
|
282
|
+
|
|
195
283
|
---
|
|
196
284
|
|
|
197
285
|
## Complete API Reference
|
|
@@ -252,6 +340,9 @@ Below is the complete reference guide for all functions available in the library
|
|
|
252
340
|
| `remove_currency_symbols(x)`| `x: str` | `str` | Strips every Unicode currency symbol. |
|
|
253
341
|
| `normalize_whitespace_and_casing(x)`| `x: str` | `str` | Canonical combination of lowercasing and whitespace collapse. |
|
|
254
342
|
| `clean_text(text, remove_stopwords=False)` | `text: str` | `str` | End-to-end cleaning (HTML, URLs, emails, lower-case, contractions, punctuation, lemmatization). |
|
|
343
|
+
| `light_clean(x, html=True, urls='', emails='', mentions=None, emojis=None, normalize='NFKC', keep_newlines=False, lowercase=False)` | `x: str` | `str` | Minimal cleaning for transformer models: Unicode normalisation, zero-width/control characters, HTML and entities, URLs/emails/mentions (remove, replace or keep), optional emoji handling. Case and punctuation are kept. |
|
|
344
|
+
| `truncate_text(x, max_tokens, tokenizer=None)` | `x: str` | `str` | Keeps the first `max_tokens` tokens (words, or tokens of a HuggingFace-style tokenizer). |
|
|
345
|
+
| `chunk_text(x, max_tokens, overlap=0, tokenizer=None)` | `x: str` | `list[str]` | Splits long text into pieces of at most `max_tokens`, with optional overlap. |
|
|
255
346
|
|
|
256
347
|
---
|
|
257
348
|
|
|
@@ -282,13 +373,14 @@ Below is the complete reference guide for all functions available in the library
|
|
|
282
373
|
|
|
283
374
|
| Function | Parameters | Return Type | Description |
|
|
284
375
|
| :--- | :--- | :--- | :--- |
|
|
285
|
-
| `detect_profanity(x)` | `x: str` | `bool` |
|
|
286
|
-
| `censor_profanity(x)` | `x: str` | `str` |
|
|
287
|
-
| `detect_toxicity(x)` | `x: str` | `float` |
|
|
288
|
-
| `
|
|
376
|
+
| `detect_profanity(x, include_mild=True, extra_words=(), allow_words=())` | `x: str` | `bool` | True if the text has profanity. Matches endings (`shitty`) and leetspeak (`sh1t`); ignores `Hell's Kitchen`; `include_mild=False` skips words like `damn`. Masked words such as `f***` are not detected. |
|
|
377
|
+
| `censor_profanity(x, include_mild=True, extra_words=(), allow_words=(), mask_char='*')` | `x: str` | `str` | Masks each profane word, keeping its length and the text layout. |
|
|
378
|
+
| `detect_toxicity(x)` | `x: str` | `float` | Score from 0.0 to 1.0. Counts insults and threats aimed at people; `kill the process` and `I do not hate you` score 0. |
|
|
379
|
+
| `toxicity_report(x)` | `x: str` | `dict` | Explains the score: `score`, `matches` (word, weight, targeted) and `ignored` (negated) words. |
|
|
380
|
+
| `detect_spam_signals(x)` | `x: str` | `dict` | Flags shouting, punctuation runs, links and spam phrases (`spam_phrases`); `is_spam_candidate` needs at least two signals. Short texts like `OK` are not flagged. |
|
|
289
381
|
| `sentiment_analysis(x, method='pattern')` | `x: str` | `str` | `'pos'` / `'neg'` / `'neu'`. Default is instant; `method='naive_bayes'` is the legacy opt-in model. |
|
|
290
382
|
| `polarity_subjectivity(x)`| `x: str` | `dict` | Returns fast pattern-based sentiment polarity (-1.0 to 1.0) and subjectivity. |
|
|
291
|
-
| `emotion_detection(x)` | `x: str` | `dict` |
|
|
383
|
+
| `emotion_detection(x)` | `x: str` | `dict` | Emotion counts (joy, anger, sadness, fear). Handles endings (`loved`, `happier`) and ignores negated words (`not happy`). |
|
|
292
384
|
|
|
293
385
|
---
|
|
294
386
|
|