nlp-text-preprocessing 0.1.1__tar.gz → 0.2.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (23) hide show
  1. nlp_text_preprocessing-0.2.0/PKG-INFO +375 -0
  2. nlp_text_preprocessing-0.2.0/Readme.md +339 -0
  3. {nlp_text_preprocessing-0.1.1 → nlp_text_preprocessing-0.2.0}/nlp_text_preprocessing/__init__.py +38 -0
  4. nlp_text_preprocessing-0.2.0/nlp_text_preprocessing/all_text_preprocess.py +720 -0
  5. nlp_text_preprocessing-0.2.0/nlp_text_preprocessing.egg-info/PKG-INFO +375 -0
  6. {nlp_text_preprocessing-0.1.1 → nlp_text_preprocessing-0.2.0}/setup.py +1 -1
  7. nlp_text_preprocessing-0.2.0/tests/test_all_text_preprocess.py +213 -0
  8. nlp_text_preprocessing-0.1.1/PKG-INFO +0 -270
  9. nlp_text_preprocessing-0.1.1/Readme.md +0 -234
  10. nlp_text_preprocessing-0.1.1/nlp_text_preprocessing/all_text_preprocess.py +0 -318
  11. nlp_text_preprocessing-0.1.1/nlp_text_preprocessing.egg-info/PKG-INFO +0 -270
  12. nlp_text_preprocessing-0.1.1/tests/test_all_text_preprocess.py +0 -139
  13. {nlp_text_preprocessing-0.1.1 → nlp_text_preprocessing-0.2.0}/LICENSE +0 -0
  14. {nlp_text_preprocessing-0.1.1 → nlp_text_preprocessing-0.2.0}/MANIFEST.in +0 -0
  15. {nlp_text_preprocessing-0.1.1 → nlp_text_preprocessing-0.2.0}/architecture_diagram.png +0 -0
  16. {nlp_text_preprocessing-0.1.1 → nlp_text_preprocessing-0.2.0}/nlp_banner.png +0 -0
  17. {nlp_text_preprocessing-0.1.1 → nlp_text_preprocessing-0.2.0}/nlp_text_preprocessing/data/contractions.json +0 -0
  18. {nlp_text_preprocessing-0.1.1 → nlp_text_preprocessing-0.2.0}/nlp_text_preprocessing.egg-info/SOURCES.txt +0 -0
  19. {nlp_text_preprocessing-0.1.1 → nlp_text_preprocessing-0.2.0}/nlp_text_preprocessing.egg-info/dependency_links.txt +0 -0
  20. {nlp_text_preprocessing-0.1.1 → nlp_text_preprocessing-0.2.0}/nlp_text_preprocessing.egg-info/requires.txt +0 -0
  21. {nlp_text_preprocessing-0.1.1 → nlp_text_preprocessing-0.2.0}/nlp_text_preprocessing.egg-info/top_level.txt +0 -0
  22. {nlp_text_preprocessing-0.1.1 → nlp_text_preprocessing-0.2.0}/requirements.txt +0 -0
  23. {nlp_text_preprocessing-0.1.1 → nlp_text_preprocessing-0.2.0}/setup.cfg +0 -0
@@ -0,0 +1,375 @@
1
+ Metadata-Version: 2.4
2
+ Name: nlp_text_preprocessing
3
+ Version: 0.2.0
4
+ Summary: This is a Text Processing Package For NLP
5
+ Author: Uditya Narayan Tiwari
6
+ Author-email: tiwarimerit@gmail.com
7
+ Classifier: Programming Language :: Python :: 3
8
+ Classifier: License :: OSI Approved :: MIT License
9
+ Classifier: Operating System :: OS Independent
10
+ Requires-Python: >=3.9
11
+ Description-Content-Type: text/markdown
12
+ License-File: LICENSE
13
+ Requires-Dist: spacy
14
+ Requires-Dist: textblob
15
+ Requires-Dist: beautifulsoup4
16
+ Requires-Dist: nltk
17
+ Requires-Dist: openpyxl
18
+ Requires-Dist: SpeechRecognition==3.10.4
19
+ Requires-Dist: pyaudio==0.2.14
20
+ Requires-Dist: PrettyTable
21
+ Requires-Dist: scikit-learn
22
+ Requires-Dist: wordcloud
23
+ Requires-Dist: lxml
24
+ Requires-Dist: pandas
25
+ Requires-Dist: numpy
26
+ Requires-Dist: matplotlib
27
+ Dynamic: author
28
+ Dynamic: author-email
29
+ Dynamic: classifier
30
+ Dynamic: description
31
+ Dynamic: description-content-type
32
+ Dynamic: license-file
33
+ Dynamic: requires-dist
34
+ Dynamic: requires-python
35
+ Dynamic: summary
36
+
37
+ # NLP Text Preprocessing Utility Package
38
+
39
+ <p align="center">
40
+ <img src="https://raw.githubusercontent.com/udityamerit/Text-Processing-Package-For-Natural-Language-Processing/main/nlp_banner.png" alt="NLP Text Preprocessing Banner" width="100%" />
41
+ </p>
42
+
43
+ <p align="center">
44
+ <a href="https://pypi.org/project/nlp-text-preprocessing/">
45
+ <img src="https://img.shields.io/pypi/v/nlp-text-preprocessing" alt="PyPI Version">
46
+ </a>
47
+ <a href="https://pypi.org/project/nlp-text-preprocessing/">
48
+ <img src="https://img.shields.io/pypi/pyversions/nlp-text-preprocessing" alt="Python Version">
49
+ </a>
50
+ <a href="https://pepy.tech/projects/nlp-text-preprocessing">
51
+ <img src="https://static.pepy.tech/badge/nlp-text-preprocessing" alt="Total Downloads">
52
+ </a>
53
+ <a href="https://pepy.tech/projects/nlp-text-preprocessing">
54
+ <img src="https://static.pepy.tech/badge/nlp-text-preprocessing/month" alt="Monthly Downloads">
55
+ </a>
56
+ <a href="https://github.com/udityamerit/Text-Processing-Package-For-Natural-Language-Processing/blob/main/LICENSE">
57
+ <img src="https://img.shields.io/github/license/udityamerit/Text-Processing-Package-For-Natural-Language-Processing" alt="License">
58
+ </a>
59
+ </p>
60
+
61
+ ---
62
+
63
+ ## Table of Contents
64
+
65
+ - [Package Overview](#package-overview)
66
+ - [Key Features](#key-features)
67
+ - [Architecture Diagram](#architecture-diagram)
68
+ - [Quickstart and Setup](#quickstart-and-setup)
69
+ - [Pipeline Automation Engine](#pipeline-automation-engine)
70
+ - [Complete API Reference](#complete-api-reference)
71
+ - [1. General Feature Extraction](#1-general-feature-extraction)
72
+ - [2. Text Cleaning and Normalization](#2-text-cleaning-and-normalization)
73
+ - [3. Linguistic and Morphological Processing](#3-linguistic-and-morphological-processing)
74
+ - [4. Safety, Moderation and Sentiment](#4-safety-moderation-and-sentiment)
75
+ - [5. Visualization Suite](#5-visualization-suite)
76
+ - [Code Examples](#code-examples)
77
+ - [Basic Text Cleaning](#1-basic-text-cleaning)
78
+ - [Pipeline Engine Usage](#2-pipeline-engine-usage)
79
+ - [Pandas DataFrame Integration](#3-pandas-dataframe-integration)
80
+ - [Batch Feature Extraction](#4-batch-feature-extraction)
81
+ - [WordCloud Generation](#5-wordcloud-generation)
82
+ - [Troubleshooting and Resource Setup](#troubleshooting-and-resource-setup)
83
+ - [License and Author](#license-and-author)
84
+
85
+ ---
86
+
87
+ ## Package Overview
88
+
89
+ **`nlp_text_preprocessing`** is a production-grade, modular Python library designed to streamline text cleaning, feature extraction, linguistic analysis, content moderation, and visualization for Natural Language Processing (NLP) pipelines and Machine Learning workflows.
90
+
91
+ It provides over **60 single-purpose functions** and an extensible **Pipeline Engine** with built-in defensive guards for `None`, `NaN`, and missing values, making it 100% crash-proof when running over Pandas DataFrames or large raw text datasets.
92
+
93
+ ---
94
+
95
+ ## Key Features
96
+
97
+ - **Crash-Proof Pipeline**: Automatic `None`/`NaN` input sanitation across all functions.
98
+ - **Pipeline Automation Engine**: Chain cleaning steps by name or custom functions via `Pipeline(['to_lower_case', 'remove_urls', 'lemmatize'])`.
99
+ - **Lazy and Fast Model Caching**: Efficient lazy-loading for spaCy (`en_core_web_sm`) and cached NaiveBayes sentiment analyzers.
100
+ - **Advanced Cleaning**: Remove emojis, phone numbers, PII mask, URLs, emails, HTML, retweets, special symbols, and expand contractions/hashtags.
101
+ - **Feature & Readability Extraction**: Extract word counts, sentence counts, readability scores (Flesch Ease), punctuation density, capital ratios, and full batch statistics.
102
+ - **Linguistic Utilities**: Named Entity Recognition (NER), POS tagging, dependency parsing, language detection, keyword extraction, synonyms, antonyms, and n-grams.
103
+ - **Safety & Moderation**: Profanity detection/censoring, toxicity scores, and spam signal detection.
104
+ - **Visualization**: Generate and save custom WordClouds to disk or display interactively.
105
+
106
+ ---
107
+
108
+ ## Architecture Diagram
109
+
110
+ The package integrates **spaCy**, **NLTK**, **TextBlob**, **BeautifulSoup4**, and **WordCloud** through a unified API surface:
111
+
112
+ <p align="center">
113
+ <img src="https://raw.githubusercontent.com/udityamerit/Text-Processing-Package-For-Natural-Language-Processing/main/architecture_diagram.png" alt="NLP Text Preprocessing Architecture Diagram" width="100%" />
114
+ </p>
115
+
116
+ ---
117
+
118
+ ## Quickstart and Setup
119
+
120
+ ### 1. Installation
121
+
122
+ ```bash
123
+ pip install nlp-text-preprocessing
124
+ ```
125
+
126
+ ### 2. Downloading Required Corpora and Models
127
+
128
+ To automatically download all required NLTK corpora (`stopwords`, `movie_reviews`, `brown`) and spaCy models (`en_core_web_sm`), run:
129
+
130
+ ```python
131
+ import nlp_text_preprocessing as tp
132
+
133
+ # One-time automated setup for NLTK and spaCy resources
134
+ tp.download_nltk_packages()
135
+ ```
136
+
137
+ ---
138
+
139
+ ## Pipeline Automation Engine
140
+
141
+ The `Pipeline` class allows chaining cleaning and transformation steps cleanly:
142
+
143
+ ```python
144
+ import nlp_text_preprocessing as tp
145
+
146
+ # 1. Create default zero-config pipeline
147
+ pipe = tp.Pipeline.default()
148
+
149
+ # 2. Or build a custom step pipeline
150
+ pipe = tp.Pipeline([
151
+ 'to_lower_case',
152
+ 'expand_hashtags',
153
+ 'remove_urls',
154
+ 'remove_emails',
155
+ 'remove_extra_whitespace',
156
+ 'lemmatize'
157
+ ])
158
+
159
+ # Process single text
160
+ clean_result = pipe.run("Check out #MachineLearning at https://example.com!")
161
+
162
+ # Process batch list or Pandas Series
163
+ clean_batch = pipe.run_batch(["Text 1", "Text 2"])
164
+ clean_series = pipe.run_series(df['text_column'])
165
+
166
+ # Generate diff report of changes
167
+ report = tp.diff_report(raw_texts, clean_batch)
168
+ ```
169
+
170
+ ---
171
+
172
+ ## Complete API Reference
173
+
174
+ Below is the complete reference guide for all functions available in the library:
175
+
176
+ ### 1. General Feature Extraction
177
+
178
+ | Function | Parameters | Return Type | Description |
179
+ | :--- | :--- | :--- | :--- |
180
+ | `word_count(x)` | `x: str` | `int` | Returns total whitespace-separated word count. |
181
+ | `char_count(x)` | `x: str` | `int` | Returns total characters excluding spaces. |
182
+ | `avg_word_len(x)` | `x: str` | `float` | Returns average word length (`0.0` for empty inputs). |
183
+ | `stop_words_count(x)` | `x: str` | `int` | Returns count of stop words (case-insensitive). |
184
+ | `sentence_count(x)` | `x: str` | `int` | Returns total number of sentences. |
185
+ | `avg_sentence_len(x)` | `x: str` | `float` | Returns average sentence length in words. |
186
+ | `punctuation_density(x)`| `x: str` | `float` | Returns proportion of punctuation characters in text. |
187
+ | `capital_ratio(x)` | `x: str` | `float` | Returns proportion of uppercase words relative to total words. |
188
+ | `unique_word_ratio(x)` | `x: str` | `float` | Returns lexical diversity ratio (unique words / total words). |
189
+ | `readability_score(x)` | `x: str` | `float` | Calculates Flesch Reading Ease score. |
190
+ | `hashtags_count(x)` | `x: str` | `int` | Returns count of hashtag tokens (`#tag`). |
191
+ | `mentions_count(x)` | `x: str` | `int` | Returns count of user handle mentions (`@user`). |
192
+ | `numerics_count(x)` | `x: str` | `int` | Returns count of numeric tokens. |
193
+ | `upper_case_count(x)` | `x: str` | `int` | Returns count of uppercase words. |
194
+ | `text_stats(x)` | `x: str` | `dict` | Returns a dictionary containing all extracted feature statistics at once. |
195
+ | `text_stats_batch(df, col)`| `df, col` | `DataFrame` | Returns a DataFrame of stats for a pandas DataFrame column. |
196
+
197
+ ---
198
+
199
+ ### 2. Text Cleaning and Normalization
200
+
201
+ | Function | Parameters | Return Type | Description |
202
+ | :--- | :--- | :--- | :--- |
203
+ | `to_lower_case(x)` | `x: str` | `str` | Converts input text to lowercase. |
204
+ | `contraction_to_expansion(x)`| `x: str` | `str` | Expands contractions using contractions dictionary. |
205
+ | `remove_emails(x)` | `x: str` | `str` | Removes email addresses from text. |
206
+ | `count_emails(x)` | `x: str` | `int` | Counts email addresses present in text. |
207
+ | `remove_urls(x)` | `x: str` | `str` | Removes HTTP/HTTPS links and `www` URLs. |
208
+ | `count_urls(x)` | `x: str` | `int` | Counts URLs present in text. |
209
+ | `remove_rt(x)` | `x: str` | `str` | Removes retweet headers (`RT @user`). |
210
+ | `count_rt(x)` | `x: str` | `int` | Counts retweet occurrences in text. |
211
+ | `remove_html_tag(x)` | `x: str` | `str` | Strips HTML/XML tags using BeautifulSoup. |
212
+ | `remove_accented_chars(x)` | `x: str` | `str` | Normalizes accented characters (NFKD -> ASCII). |
213
+ | `remove_mentions(x)` | `x: str` | `str` | Removes user handle mentions (`@username`). |
214
+ | `remove_special_chars(x)` | `x: str` | `str` | Strips punctuation and special symbols. |
215
+ | `remove_repeated_chars(x)` | `x: str` | `str` | Truncates repeated characters beyond 2 consecutive occurrences. |
216
+ | `remove_stop_words(x)` | `x: str` | `str` | Case-insensitive removal of stop words while preserving word case. |
217
+ | `remove_emojis(x)` | `x: str` | `str` | Strips emoji characters from text. |
218
+ | `extract_emojis(x)` | `x: str` | `list[str]` | Returns list of emojis found in text. |
219
+ | `emoji_to_text(x)` | `x: str` | `str` | Converts emojis to descriptive text (e.g. 😂 -> `face_with_tears_of_joy`). |
220
+ | `remove_phone_numbers(x)` | `x: str` | `str` | Strips phone numbers from text. |
221
+ | `count_phone_numbers(x)` | `x: str` | `int` | Counts phone numbers present in text. |
222
+ | `mask_pii(x)` | `x: str` | `str` | Redacts emails, phone numbers, and card numbers with `[REDACTED]`. |
223
+ | `remove_extra_whitespace(x)`| `x: str` | `str` | Collapses consecutive spaces, tabs, and newlines to a single space. |
224
+ | `expand_hashtags(x)` | `x: str` | `str` | Splits camelCase/PascalCase hashtags (`#MachineLearning` -> `Machine Learning`). |
225
+ | `normalize_unicode_punctuation(x)`| `x: str` | `str` | Replaces smart quotes, em-dashes, and ellipsis with ASCII equivalents. |
226
+ | `remove_numbers(x)` | `x: str` | `str` | Strips all numeric digits. |
227
+ | `remove_currency_symbols(x)`| `x: str` | `str` | Strips currency symbols (`$`, `€`, `₹`, `£`, `¥`). |
228
+ | `normalize_whitespace_and_casing(x)`| `x: str` | `str` | Canonical combination of lowercasing and whitespace collapse. |
229
+ | `clean_text(text)` | `text: str` | `str` | Runs end-to-end cleaning pipeline. |
230
+
231
+ ---
232
+
233
+ ### 3. Linguistic and Morphological Processing
234
+
235
+ | Function | Parameters | Return Type | Description |
236
+ | :--- | :--- | :--- | :--- |
237
+ | `convert_to_base(x)` | `x: str` | `str` | Lemmatizes nouns and verbs via spaCy while keeping other POS tags. |
238
+ | `lemmatize(x)` | `x: str` | `str` | Performs full lemmatization across all tokens via spaCy. |
239
+ | `extract_entities(x)` | `x: str` | `list[tuple]` | Extracts named entities `(text, label)` using spaCy. |
240
+ | `extract_pos_tags(x)` | `x: str` | `list[tuple]` | Extracts full POS tag breakdown `(token, pos, tag)` using spaCy. |
241
+ | `get_dependency_tree(x)`| `x: str` | `list[tuple]` | Extracts dependency parse tree `(token, dep, head)` using spaCy. |
242
+ | `detect_language(x)` | `x: str` | `str` | Detects language code (`'en'`, `'fr'`, etc.) using TextBlob. |
243
+ | `keyword_extraction(x, top_n)`| `x: str, top_n: int`| `list[str]`| Extracts top N keywords by term frequency. |
244
+ | `text_similarity(x1, x2)` | `x1, x2` | `float` | Calculates token Jaccard similarity between two texts (0.0 to 1.0). |
245
+ | `get_synonyms(word)` | `word: str` | `list[str]` | Returns list of synonyms from WordNet. |
246
+ | `get_antonyms(word)` | `word: str` | `list[str]` | Returns list of antonyms from WordNet. |
247
+ | `chunk_noun_phrases(x)` | `x: str` | `list[dict]` | Extracts noun phrases with character start/end spans. |
248
+ | `correct_spelling(x)` | `x: str` | `str` | Corrects word spelling using TextBlob model. |
249
+ | `get_noun_phrase(x)` | `x: str` | `list[str]` | Extracts noun phrases from text. |
250
+ | `n_grams(x, n=2)` | `x: str, n: int` | `list` | Generates word-level n-grams. |
251
+ | `singularize_words(x)` | `x: str` | `str` | Converts plural nouns (`NNS`) into singular forms. |
252
+ | `pluralize_words(x)` | `x: str` | `str` | Converts singular nouns (`NN`) into plural forms. |
253
+
254
+ ---
255
+
256
+ ### 4. Safety, Moderation and Sentiment
257
+
258
+ | Function | Parameters | Return Type | Description |
259
+ | :--- | :--- | :--- | :--- |
260
+ | `detect_profanity(x)` | `x: str` | `bool` | Returns True if text contains profanity words. |
261
+ | `censor_profanity(x)` | `x: str` | `str` | Censors profanity words with asterisks (`****`). |
262
+ | `detect_toxicity(x)` | `x: str` | `float` | Calculates baseline toxicity score (0.0 to 1.0). |
263
+ | `detect_spam_signals(x)` | `x: str` | `dict` | Returns spam flags for excessive caps, punctuation, and URL density. |
264
+ | `sentiment_analysis(x)` | `x: str` | `str` | Returns sentiment classification (`'pos'` / `'neg'`) using cached NaiveBayes model. |
265
+ | `polarity_subjectivity(x)`| `x: str` | `dict` | Returns fast pattern-based sentiment polarity (-1.0 to 1.0) and subjectivity. |
266
+ | `emotion_detection(x)` | `x: str` | `dict` | Calculates lexicon-based emotion counts (joy, anger, sadness, fear). |
267
+
268
+ ---
269
+
270
+ ### 5. Visualization Suite
271
+
272
+ | Function | Parameters | Return Type | Description |
273
+ | :--- | :--- | :--- | :--- |
274
+ | `get_wordcloud(x, save_path=None, show=True)` | `x: str, save_path: str, show: bool` | `WordCloud` | Generates a WordCloud object, with optional file saving and GUI toggle options. |
275
+
276
+ ---
277
+
278
+ ## Code Examples
279
+
280
+ ### 1. Basic Text Cleaning
281
+
282
+ ```python
283
+ import nlp_text_preprocessing as tp
284
+
285
+ raw_text = "Check out https://example.com! Contact support@company.org or RT @user 'I'm loving #NLP'."
286
+ cleaned = tp.clean_text(raw_text)
287
+
288
+ print(cleaned)
289
+ # Output: check out contact support company org or i am love nlp
290
+ ```
291
+
292
+ ### 2. Pipeline Engine Usage
293
+
294
+ ```python
295
+ import nlp_text_preprocessing as tp
296
+
297
+ # Build custom pipeline
298
+ pipe = tp.Pipeline(['to_lower_case', 'expand_hashtags', 'remove_urls', 'remove_extra_whitespace'])
299
+
300
+ text = "Check out #MachineLearning at https://example.com!"
301
+ print(pipe.run(text))
302
+ # Output: check out machine learning at!
303
+ ```
304
+
305
+ ### 3. Pandas DataFrame Integration
306
+
307
+ All functions seamlessly support Pandas `.apply()` and handle missing values (`NaN`/`None`):
308
+
309
+ ```python
310
+ import pandas as pd
311
+ import nlp_text_preprocessing as tp
312
+
313
+ df = pd.DataFrame({
314
+ 'text': [
315
+ "I'm loving this NLP package! #awesome",
316
+ "Contact me at info@test.com or visit https://test.com",
317
+ None
318
+ ]
319
+ })
320
+
321
+ # Apply end-to-end text cleaning
322
+ df['clean_text'] = df['text'].apply(tp.clean_text)
323
+
324
+ # Extract word counts safely
325
+ df['word_count'] = df['text'].apply(tp.word_count)
326
+ ```
327
+
328
+ ### 4. Batch Feature Extraction
329
+
330
+ Extract full feature matrices directly into Pandas columns:
331
+
332
+ ```python
333
+ import pandas as pd
334
+ import nlp_text_preprocessing as tp
335
+
336
+ df = pd.DataFrame({'text': ["Hello #world! Visit https://example.com", "Contact user@test.com"]})
337
+
338
+ # Extract all summary stats into a new DataFrame
339
+ stats_df = tp.text_stats_batch(df, 'text')
340
+ print(stats_df)
341
+ ```
342
+
343
+ ### 5. WordCloud Generation
344
+
345
+ ```python
346
+ import nlp_text_preprocessing as tp
347
+
348
+ text = "python natural language processing machine learning text mining data science spaCy nltk textblob"
349
+
350
+ # Save directly to disk without opening interactive GUI window
351
+ tp.get_wordcloud(text, save_path="wordcloud.png", show=False)
352
+ ```
353
+
354
+ ---
355
+
356
+ ## Troubleshooting and Resource Setup
357
+
358
+ If any NLTK corpus or spaCy model throws a missing resource error during runtime, simply run:
359
+
360
+ ```python
361
+ import nlp_text_preprocessing as tp
362
+ tp.download_nltk_packages()
363
+ ```
364
+
365
+ Or manually download spaCy model via terminal:
366
+ ```bash
367
+ python -m spacy download en_core_web_sm
368
+ ```
369
+
370
+ ---
371
+
372
+ ## License and Author
373
+
374
+ - **Author**: [Uditya Narayan Tiwari](https://youtube.com/kgptalkie)
375
+ - **License**: MIT License ([LICENSE](LICENSE))