nlp-text-preprocessing 0.3.0__tar.gz → 0.4.1__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (30) hide show
  1. {nlp_text_preprocessing-0.3.0 → nlp_text_preprocessing-0.4.1}/PKG-INFO +99 -7
  2. nlp_text_preprocessing-0.3.0/nlp_text_preprocessing.egg-info/PKG-INFO → nlp_text_preprocessing-0.4.1/Readme.md +435 -401
  3. {nlp_text_preprocessing-0.3.0 → nlp_text_preprocessing-0.4.1}/nlp_text_preprocessing/__init__.py +14 -1
  4. nlp_text_preprocessing-0.4.1/nlp_text_preprocessing/__main__.py +5 -0
  5. {nlp_text_preprocessing-0.3.0 → nlp_text_preprocessing-0.4.1}/nlp_text_preprocessing/all_text_preprocess.py +463 -51
  6. nlp_text_preprocessing-0.4.1/nlp_text_preprocessing/cli.py +150 -0
  7. nlp_text_preprocessing-0.4.1/nlp_text_preprocessing/sklearn_compat.py +44 -0
  8. nlp_text_preprocessing-0.3.0/Readme.md → nlp_text_preprocessing-0.4.1/nlp_text_preprocessing.egg-info/PKG-INFO +493 -349
  9. {nlp_text_preprocessing-0.3.0 → nlp_text_preprocessing-0.4.1}/nlp_text_preprocessing.egg-info/SOURCES.txt +9 -1
  10. nlp_text_preprocessing-0.4.1/nlp_text_preprocessing.egg-info/entry_points.txt +2 -0
  11. {nlp_text_preprocessing-0.3.0 → nlp_text_preprocessing-0.4.1}/nlp_text_preprocessing.egg-info/requires.txt +8 -0
  12. {nlp_text_preprocessing-0.3.0 → nlp_text_preprocessing-0.4.1}/pyproject.toml +7 -2
  13. nlp_text_preprocessing-0.4.1/requirements.txt +15 -0
  14. {nlp_text_preprocessing-0.3.0 → nlp_text_preprocessing-0.4.1}/tests/test_infrastructure.py +1 -1
  15. nlp_text_preprocessing-0.4.1/tests/test_known_limits.py +38 -0
  16. nlp_text_preprocessing-0.4.1/tests/test_light_and_cli.py +195 -0
  17. nlp_text_preprocessing-0.4.1/tests/test_moderation.py +102 -0
  18. nlp_text_preprocessing-0.4.1/tests/test_pipeline_api.py +100 -0
  19. nlp_text_preprocessing-0.3.0/requirements.txt +0 -7
  20. {nlp_text_preprocessing-0.3.0 → nlp_text_preprocessing-0.4.1}/LICENSE +0 -0
  21. {nlp_text_preprocessing-0.3.0 → nlp_text_preprocessing-0.4.1}/MANIFEST.in +0 -0
  22. {nlp_text_preprocessing-0.3.0 → nlp_text_preprocessing-0.4.1}/architecture_diagram.png +0 -0
  23. {nlp_text_preprocessing-0.3.0 → nlp_text_preprocessing-0.4.1}/nlp_banner.png +0 -0
  24. {nlp_text_preprocessing-0.3.0 → nlp_text_preprocessing-0.4.1}/nlp_text_preprocessing/data/contractions.json +0 -0
  25. {nlp_text_preprocessing-0.3.0 → nlp_text_preprocessing-0.4.1}/nlp_text_preprocessing.egg-info/dependency_links.txt +0 -0
  26. {nlp_text_preprocessing-0.3.0 → nlp_text_preprocessing-0.4.1}/nlp_text_preprocessing.egg-info/top_level.txt +0 -0
  27. {nlp_text_preprocessing-0.3.0 → nlp_text_preprocessing-0.4.1}/setup.cfg +0 -0
  28. {nlp_text_preprocessing-0.3.0 → nlp_text_preprocessing-0.4.1}/tests/test_all_text_preprocess.py +0 -0
  29. {nlp_text_preprocessing-0.3.0 → nlp_text_preprocessing-0.4.1}/tests/test_cleaning.py +0 -0
  30. {nlp_text_preprocessing-0.3.0 → nlp_text_preprocessing-0.4.1}/tests/test_linguistic.py +0 -0
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: nlp-text-preprocessing
3
- Version: 0.3.0
3
+ Version: 0.4.1
4
4
  Summary: Text cleaning, feature extraction and linguistic utilities for NLP and ML pipelines
5
5
  Author-email: Uditya Narayan Tiwari <tiwarimerit@gmail.com>
6
6
  License-Expression: MIT
@@ -37,10 +37,15 @@ Requires-Dist: pandas; extra == "pandas"
37
37
  Provides-Extra: viz
38
38
  Requires-Dist: wordcloud; extra == "viz"
39
39
  Requires-Dist: matplotlib; extra == "viz"
40
+ Provides-Extra: sklearn
41
+ Requires-Dist: scikit-learn; extra == "sklearn"
42
+ Provides-Extra: transformers
43
+ Requires-Dist: transformers; extra == "transformers"
40
44
  Provides-Extra: all
41
45
  Requires-Dist: pandas; extra == "all"
42
46
  Requires-Dist: wordcloud; extra == "all"
43
47
  Requires-Dist: matplotlib; extra == "all"
48
+ Requires-Dist: scikit-learn; extra == "all"
44
49
  Provides-Extra: dev
45
50
  Requires-Dist: pytest; extra == "dev"
46
51
  Requires-Dist: build; extra == "dev"
@@ -48,6 +53,7 @@ Requires-Dist: twine; extra == "dev"
48
53
  Requires-Dist: pandas; extra == "dev"
49
54
  Requires-Dist: wordcloud; extra == "dev"
50
55
  Requires-Dist: matplotlib; extra == "dev"
56
+ Requires-Dist: scikit-learn; extra == "dev"
51
57
  Dynamic: license-file
52
58
 
53
59
  # NLP Text Preprocessing Utility Package
@@ -83,6 +89,8 @@ Dynamic: license-file
83
89
  - [Architecture Diagram](#architecture-diagram)
84
90
  - [Quickstart and Setup](#quickstart-and-setup)
85
91
  - [Pipeline Automation Engine](#pipeline-automation-engine)
92
+ - [Preparing Text for Transformers](#preparing-text-for-transformers-bert-gpt-)
93
+ - [Command Line (`nlp-prep`)](#command-line-nlp-prep)
86
94
  - [Complete API Reference](#complete-api-reference)
87
95
  - [1. General Feature Extraction](#1-general-feature-extraction)
88
96
  - [2. Text Cleaning and Normalization](#2-text-cleaning-and-normalization)
@@ -118,7 +126,7 @@ It provides over **60 single-purpose functions** and an extensible **Pipeline En
118
126
  - **Advanced Cleaning**: Remove emojis, phone numbers, PII mask, URLs, emails, HTML, retweets, special symbols, and expand contractions/hashtags.
119
127
  - **Feature & Readability Extraction**: Extract word counts, sentence counts, readability scores (Flesch Ease), punctuation density, capital ratios, and full batch statistics.
120
128
  - **Linguistic Utilities**: Named Entity Recognition (NER), POS tagging, dependency parsing, language detection, keyword extraction, synonyms, antonyms, and n-grams.
121
- - **Safety & Moderation**: Profanity detection/censoring, toxicity scores, and spam signal detection.
129
+ - **Safety & Moderation (baseline)**: Lexicon-based profanity, toxicity, spam and emotion helpers. They are transparent keyword baselines, not a moderation system: use a trained model for anything high-stakes.
122
130
  - **Visualization**: Generate and save custom WordClouds to disk or display interactively.
123
131
 
124
132
  ---
@@ -141,6 +149,7 @@ The package integrates **spaCy**, **NLTK**, **TextBlob**, **BeautifulSoup4**, an
141
149
  pip install nlp-text-preprocessing # core
142
150
  pip install "nlp-text-preprocessing[viz]" # + wordcloud, matplotlib (get_wordcloud)
143
151
  pip install "nlp-text-preprocessing[pandas]" # + pandas (text_stats_batch, Pipeline.run_series)
152
+ pip install "nlp-text-preprocessing[sklearn]" # + scikit-learn (TextPreprocessor)
144
153
  pip install "nlp-text-preprocessing[all]" # everything optional
145
154
  ```
146
155
 
@@ -192,6 +201,85 @@ clean_series = pipe.run_series(df['text_column'])
192
201
  report = tp.diff_report(raw_texts, clean_batch)
193
202
  ```
194
203
 
204
+ ### Steps with arguments, saving and scikit-learn
205
+
206
+ ```python
207
+ # Pass arguments as (name, {argument: value})
208
+ pipe = tp.Pipeline([
209
+ 'to_lower_case',
210
+ ('remove_stop_words', {'keep_negations': False}),
211
+ ])
212
+ print(pipe) # Pipeline(['to_lower_case', "remove_stop_words{'keep_negations': False}"])
213
+
214
+ # Save and reload (custom functions must be registered with @tp.register_step('name'))
215
+ pipe.save('my_pipeline.json')
216
+ pipe = tp.Pipeline.load('my_pipeline.json')
217
+
218
+ # Use it inside scikit-learn: pip install "nlp-text-preprocessing[sklearn]"
219
+ from sklearn.pipeline import make_pipeline
220
+ from sklearn.feature_extraction.text import TfidfVectorizer
221
+
222
+ model = make_pipeline(tp.TextPreprocessor(['to_lower_case', 'remove_special_chars']),
223
+ TfidfVectorizer())
224
+ X = model.fit_transform(["Hello, World!", "HELLO there"])
225
+ ```
226
+
227
+ Every step except the last must return a string. The last step may return anything,
228
+ for example `Pipeline(['to_lower_case', 'text_stats'])`.
229
+
230
+ ---
231
+
232
+ ## Preparing Text for Transformers (BERT, GPT, ...)
233
+
234
+ Modern models work best with *lightly* cleaned text: keep the case, punctuation and
235
+ word forms, and only remove noise. `light_clean` does exactly that, and `truncate_text`
236
+ / `chunk_text` help you stay inside a model's token limit.
237
+
238
+ ```python
239
+ import nlp_text_preprocessing as tp
240
+
241
+ tp.light_clean("<p>Great &amp; fast!​ Visit https://x.com or mail me@x.com</p>",
242
+ urls='[URL]', emails='[EMAIL]')
243
+ # 'Great & fast! Visit [URL] or mail [EMAIL]'
244
+
245
+ # Use the real tokenizer so the limits are counted in model tokens
246
+ # (pip install "nlp-text-preprocessing[transformers]")
247
+ from transformers import AutoTokenizer
248
+ tok = AutoTokenizer.from_pretrained('bert-base-uncased')
249
+
250
+ tp.truncate_text(long_text, 256, tokenizer=tok)
251
+ chunks = tp.chunk_text(long_text, 256, overlap=32, tokenizer=tok)
252
+ ```
253
+
254
+ Without a `tokenizer`, the limits are counted in whitespace-separated words.
255
+ `tokenizer` may be a HuggingFace tokenizer (anything with `.tokenize()` and
256
+ `.convert_tokens_to_string()`) or a plain function that returns a list of tokens.
257
+
258
+ ---
259
+
260
+ ## Command Line (`nlp-prep`)
261
+
262
+ Installing the package also installs the `nlp-prep` command (`python -m nlp_text_preprocessing` works too).
263
+
264
+ ```bash
265
+ # one document per line, written to a new file
266
+ nlp-prep clean reviews.txt -o reviews_clean.txt
267
+
268
+ # a CSV column (adds a 'text_clean' column; use --output-column text to overwrite)
269
+ nlp-prep clean data.csv --column text -o data_clean.csv
270
+
271
+ # standard input / output
272
+ cat notes.txt | nlp-prep clean - --lowercase --url-token "[URL]"
273
+
274
+ # classic cleaning, or your own steps / a saved Pipeline
275
+ nlp-prep clean reviews.txt --mode standard --stopwords
276
+ nlp-prep clean reviews.txt --steps to_lower_case,remove_urls
277
+ nlp-prep clean reviews.txt --pipeline my_pipeline.json
278
+
279
+ # check or install the NLTK data and spaCy model
280
+ nlp-prep resources --download
281
+ ```
282
+
195
283
  ---
196
284
 
197
285
  ## Complete API Reference
@@ -252,6 +340,9 @@ Below is the complete reference guide for all functions available in the library
252
340
  | `remove_currency_symbols(x)`| `x: str` | `str` | Strips every Unicode currency symbol. |
253
341
  | `normalize_whitespace_and_casing(x)`| `x: str` | `str` | Canonical combination of lowercasing and whitespace collapse. |
254
342
  | `clean_text(text, remove_stopwords=False)` | `text: str` | `str` | End-to-end cleaning (HTML, URLs, emails, lower-case, contractions, punctuation, lemmatization). |
343
+ | `light_clean(x, html=True, urls='', emails='', mentions=None, emojis=None, normalize='NFKC', keep_newlines=False, lowercase=False)` | `x: str` | `str` | Minimal cleaning for transformer models: Unicode normalisation, zero-width/control characters, HTML and entities, URLs/emails/mentions (remove, replace or keep), optional emoji handling. Case and punctuation are kept. |
344
+ | `truncate_text(x, max_tokens, tokenizer=None)` | `x: str` | `str` | Keeps the first `max_tokens` tokens (words, or tokens of a HuggingFace-style tokenizer). |
345
+ | `chunk_text(x, max_tokens, overlap=0, tokenizer=None)` | `x: str` | `list[str]` | Splits long text into pieces of at most `max_tokens`, with optional overlap. |
255
346
 
256
347
  ---
257
348
 
@@ -282,13 +373,14 @@ Below is the complete reference guide for all functions available in the library
282
373
 
283
374
  | Function | Parameters | Return Type | Description |
284
375
  | :--- | :--- | :--- | :--- |
285
- | `detect_profanity(x)` | `x: str` | `bool` | Returns True if text contains profanity words. |
286
- | `censor_profanity(x)` | `x: str` | `str` | Censors profanity words with asterisks (`****`). |
287
- | `detect_toxicity(x)` | `x: str` | `float` | Calculates baseline toxicity score (0.0 to 1.0). |
288
- | `detect_spam_signals(x)` | `x: str` | `dict` | Returns spam flags for excessive caps, punctuation, and URL density. |
376
+ | `detect_profanity(x, include_mild=True, extra_words=(), allow_words=())` | `x: str` | `bool` | True if the text has profanity. Matches endings (`shitty`) and leetspeak (`sh1t`); ignores `Hell's Kitchen`; `include_mild=False` skips words like `damn`. Masked words such as `f***` are not detected. |
377
+ | `censor_profanity(x, include_mild=True, extra_words=(), allow_words=(), mask_char='*')` | `x: str` | `str` | Masks each profane word, keeping its length and the text layout. |
378
+ | `detect_toxicity(x)` | `x: str` | `float` | Score from 0.0 to 1.0. Counts insults and threats aimed at people; `kill the process` and `I do not hate you` score 0. |
379
+ | `toxicity_report(x)` | `x: str` | `dict` | Explains the score: `score`, `matches` (word, weight, targeted) and `ignored` (negated) words. |
380
+ | `detect_spam_signals(x)` | `x: str` | `dict` | Flags shouting, punctuation runs, links and spam phrases (`spam_phrases`); `is_spam_candidate` needs at least two signals. Short texts like `OK` are not flagged. |
289
381
  | `sentiment_analysis(x, method='pattern')` | `x: str` | `str` | `'pos'` / `'neg'` / `'neu'`. Default is instant; `method='naive_bayes'` is the legacy opt-in model. |
290
382
  | `polarity_subjectivity(x)`| `x: str` | `dict` | Returns fast pattern-based sentiment polarity (-1.0 to 1.0) and subjectivity. |
291
- | `emotion_detection(x)` | `x: str` | `dict` | Calculates lexicon-based emotion counts (joy, anger, sadness, fear). |
383
+ | `emotion_detection(x)` | `x: str` | `dict` | Emotion counts (joy, anger, sadness, fear). Handles endings (`loved`, `happier`) and ignores negated words (`not happy`). |
292
384
 
293
385
  ---
294
386