nlp-text-preprocessing 0.1.2__tar.gz → 0.2.1__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (21) hide show
  1. {nlp_text_preprocessing-0.1.2 → nlp_text_preprocessing-0.2.1}/PKG-INFO +122 -22
  2. nlp_text_preprocessing-0.1.2/nlp_text_preprocessing.egg-info/PKG-INFO → nlp_text_preprocessing-0.2.1/Readme.md +341 -283
  3. {nlp_text_preprocessing-0.1.2 → nlp_text_preprocessing-0.2.1}/nlp_text_preprocessing/__init__.py +38 -0
  4. nlp_text_preprocessing-0.2.1/nlp_text_preprocessing/all_text_preprocess.py +720 -0
  5. nlp_text_preprocessing-0.2.1/nlp_text_preprocessing.egg-info/PKG-INFO +383 -0
  6. {nlp_text_preprocessing-0.1.2 → nlp_text_preprocessing-0.2.1}/setup.py +7 -1
  7. nlp_text_preprocessing-0.2.1/tests/test_all_text_preprocess.py +213 -0
  8. nlp_text_preprocessing-0.1.2/Readme.md +0 -247
  9. nlp_text_preprocessing-0.1.2/nlp_text_preprocessing/all_text_preprocess.py +0 -318
  10. nlp_text_preprocessing-0.1.2/tests/test_all_text_preprocess.py +0 -139
  11. {nlp_text_preprocessing-0.1.2 → nlp_text_preprocessing-0.2.1}/LICENSE +0 -0
  12. {nlp_text_preprocessing-0.1.2 → nlp_text_preprocessing-0.2.1}/MANIFEST.in +0 -0
  13. {nlp_text_preprocessing-0.1.2 → nlp_text_preprocessing-0.2.1}/architecture_diagram.png +0 -0
  14. {nlp_text_preprocessing-0.1.2 → nlp_text_preprocessing-0.2.1}/nlp_banner.png +0 -0
  15. {nlp_text_preprocessing-0.1.2 → nlp_text_preprocessing-0.2.1}/nlp_text_preprocessing/data/contractions.json +0 -0
  16. {nlp_text_preprocessing-0.1.2 → nlp_text_preprocessing-0.2.1}/nlp_text_preprocessing.egg-info/SOURCES.txt +0 -0
  17. {nlp_text_preprocessing-0.1.2 → nlp_text_preprocessing-0.2.1}/nlp_text_preprocessing.egg-info/dependency_links.txt +0 -0
  18. {nlp_text_preprocessing-0.1.2 → nlp_text_preprocessing-0.2.1}/nlp_text_preprocessing.egg-info/requires.txt +0 -0
  19. {nlp_text_preprocessing-0.1.2 → nlp_text_preprocessing-0.2.1}/nlp_text_preprocessing.egg-info/top_level.txt +0 -0
  20. {nlp_text_preprocessing-0.1.2 → nlp_text_preprocessing-0.2.1}/requirements.txt +0 -0
  21. {nlp_text_preprocessing-0.1.2 → nlp_text_preprocessing-0.2.1}/setup.cfg +0 -0
@@ -1,9 +1,13 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: nlp_text_preprocessing
3
- Version: 0.1.2
3
+ Version: 0.2.1
4
4
  Summary: This is a Text Processing Package For NLP
5
+ Home-page: https://github.com/udityamerit/Text-Processing-Package-For-Natural-Language-Processing
5
6
  Author: Uditya Narayan Tiwari
6
7
  Author-email: tiwarimerit@gmail.com
8
+ Project-URL: Author Website, https://udityanarayantiwari.netlify.app/
9
+ Project-URL: Source Code, https://github.com/udityamerit/Text-Processing-Package-For-Natural-Language-Processing
10
+ Project-URL: Bug Tracker, https://github.com/udityamerit/Text-Processing-Package-For-Natural-Language-Processing/issues
7
11
  Classifier: Programming Language :: Python :: 3
8
12
  Classifier: License :: OSI Approved :: MIT License
9
13
  Classifier: Operating System :: OS Independent
@@ -29,7 +33,9 @@ Dynamic: author-email
29
33
  Dynamic: classifier
30
34
  Dynamic: description
31
35
  Dynamic: description-content-type
36
+ Dynamic: home-page
32
37
  Dynamic: license-file
38
+ Dynamic: project-url
33
39
  Dynamic: requires-dist
34
40
  Dynamic: requires-python
35
41
  Dynamic: summary
@@ -66,16 +72,19 @@ Dynamic: summary
66
72
  - [Key Features](#key-features)
67
73
  - [Architecture Diagram](#architecture-diagram)
68
74
  - [Quickstart and Setup](#quickstart-and-setup)
75
+ - [Pipeline Automation Engine](#pipeline-automation-engine)
69
76
  - [Complete API Reference](#complete-api-reference)
70
77
  - [1. General Feature Extraction](#1-general-feature-extraction)
71
78
  - [2. Text Cleaning and Normalization](#2-text-cleaning-and-normalization)
72
79
  - [3. Linguistic and Morphological Processing](#3-linguistic-and-morphological-processing)
73
- - [4. Visualization Suite](#4-visualization-suite)
80
+ - [4. Safety, Moderation and Sentiment](#4-safety-moderation-and-sentiment)
81
+ - [5. Visualization Suite](#5-visualization-suite)
74
82
  - [Code Examples](#code-examples)
75
83
  - [Basic Text Cleaning](#1-basic-text-cleaning)
76
- - [Pandas DataFrame Integration](#2-pandas-dataframe-integration)
77
- - [Batch Feature Extraction](#3-batch-feature-extraction)
78
- - [WordCloud Generation](#4-wordcloud-generation)
84
+ - [Pipeline Engine Usage](#2-pipeline-engine-usage)
85
+ - [Pandas DataFrame Integration](#3-pandas-dataframe-integration)
86
+ - [Batch Feature Extraction](#4-batch-feature-extraction)
87
+ - [WordCloud Generation](#5-wordcloud-generation)
79
88
  - [Troubleshooting and Resource Setup](#troubleshooting-and-resource-setup)
80
89
  - [License and Author](#license-and-author)
81
90
 
@@ -83,20 +92,24 @@ Dynamic: summary
83
92
 
84
93
  ## Package Overview
85
94
 
86
- **`nlp_text_preprocessing`** is a production-grade, modular Python library designed to streamline text cleaning, feature extraction, linguistic analysis, and visualization for Natural Language Processing (NLP) pipelines and Machine Learning workflows.
95
+ **`nlp_text_preprocessing`** is a production-grade, modular Python library designed to streamline text cleaning, feature extraction, linguistic analysis, content moderation, and visualization for Natural Language Processing (NLP) pipelines and Machine Learning workflows.
87
96
 
88
- It provides **34 single-purpose functions** with built-in defensive guards for `None`, `NaN`, and missing values, making it 100% crash-proof when running over Pandas DataFrames or large raw text datasets.
97
+ Maintained by **[Uditya Narayan Tiwari](https://udityanarayantiwari.netlify.app/)**.
98
+
99
+ It provides over **60 single-purpose functions** and an extensible **Pipeline Engine** with built-in defensive guards for `None`, `NaN`, and missing values, making it 100% crash-proof when running over Pandas DataFrames or large raw text datasets.
89
100
 
90
101
  ---
91
102
 
92
103
  ## Key Features
93
104
 
94
105
  - **Crash-Proof Pipeline**: Automatic `None`/`NaN` input sanitation across all functions.
106
+ - **Pipeline Automation Engine**: Chain cleaning steps by name or custom functions via `Pipeline(['to_lower_case', 'remove_urls', 'lemmatize'])`.
95
107
  - **Lazy and Fast Model Caching**: Efficient lazy-loading for spaCy (`en_core_web_sm`) and cached NaiveBayes sentiment analyzers.
96
- - **Comprehensive Cleaning**: Strip emails, URLs, HTML, retweets, special symbols, and expand contractions.
97
- - **Feature Extraction**: Extract word counts, character counts, uppercase counts, numerics, hashtags, mentions, and full feature matrices in one call.
98
- - **Linguistic Utilities**: Perform lemmatization, noun phrase extraction, spelling correction, n-grams, singularization, and pluralization.
99
- - **Flexible Visualizations**: Generate and save custom WordClouds to disk or display interactively.
108
+ - **Advanced Cleaning**: Remove emojis, phone numbers, PII mask, URLs, emails, HTML, retweets, special symbols, and expand contractions/hashtags.
109
+ - **Feature & Readability Extraction**: Extract word counts, sentence counts, readability scores (Flesch Ease), punctuation density, capital ratios, and full batch statistics.
110
+ - **Linguistic Utilities**: Named Entity Recognition (NER), POS tagging, dependency parsing, language detection, keyword extraction, synonyms, antonyms, and n-grams.
111
+ - **Safety & Moderation**: Profanity detection/censoring, toxicity scores, and spam signal detection.
112
+ - **Visualization**: Generate and save custom WordClouds to disk or display interactively.
100
113
 
101
114
  ---
102
115
 
@@ -131,9 +144,42 @@ tp.download_nltk_packages()
131
144
 
132
145
  ---
133
146
 
147
+ ## Pipeline Automation Engine
148
+
149
+ The `Pipeline` class allows chaining cleaning and transformation steps cleanly:
150
+
151
+ ```python
152
+ import nlp_text_preprocessing as tp
153
+
154
+ # 1. Create default zero-config pipeline
155
+ pipe = tp.Pipeline.default()
156
+
157
+ # 2. Or build a custom step pipeline
158
+ pipe = tp.Pipeline([
159
+ 'to_lower_case',
160
+ 'expand_hashtags',
161
+ 'remove_urls',
162
+ 'remove_emails',
163
+ 'remove_extra_whitespace',
164
+ 'lemmatize'
165
+ ])
166
+
167
+ # Process single text
168
+ clean_result = pipe.run("Check out #MachineLearning at https://example.com!")
169
+
170
+ # Process batch list or Pandas Series
171
+ clean_batch = pipe.run_batch(["Text 1", "Text 2"])
172
+ clean_series = pipe.run_series(df['text_column'])
173
+
174
+ # Generate diff report of changes
175
+ report = tp.diff_report(raw_texts, clean_batch)
176
+ ```
177
+
178
+ ---
179
+
134
180
  ## Complete API Reference
135
181
 
136
- Below is the complete reference guide for all **34 functions** available in the library:
182
+ Below is the complete reference guide for all functions available in the library:
137
183
 
138
184
  ### 1. General Feature Extraction
139
185
 
@@ -143,11 +189,18 @@ Below is the complete reference guide for all **34 functions** available in the
143
189
  | `char_count(x)` | `x: str` | `int` | Returns total characters excluding spaces. |
144
190
  | `avg_word_len(x)` | `x: str` | `float` | Returns average word length (`0.0` for empty inputs). |
145
191
  | `stop_words_count(x)` | `x: str` | `int` | Returns count of stop words (case-insensitive). |
192
+ | `sentence_count(x)` | `x: str` | `int` | Returns total number of sentences. |
193
+ | `avg_sentence_len(x)` | `x: str` | `float` | Returns average sentence length in words. |
194
+ | `punctuation_density(x)`| `x: str` | `float` | Returns proportion of punctuation characters in text. |
195
+ | `capital_ratio(x)` | `x: str` | `float` | Returns proportion of uppercase words relative to total words. |
196
+ | `unique_word_ratio(x)` | `x: str` | `float` | Returns lexical diversity ratio (unique words / total words). |
197
+ | `readability_score(x)` | `x: str` | `float` | Calculates Flesch Reading Ease score. |
146
198
  | `hashtags_count(x)` | `x: str` | `int` | Returns count of hashtag tokens (`#tag`). |
147
199
  | `mentions_count(x)` | `x: str` | `int` | Returns count of user handle mentions (`@user`). |
148
200
  | `numerics_count(x)` | `x: str` | `int` | Returns count of numeric tokens. |
149
201
  | `upper_case_count(x)` | `x: str` | `int` | Returns count of uppercase words. |
150
- | `extract_features(x)` | `x: str` | `dict` | Returns a dictionary containing all extracted feature counts at once. |
202
+ | `text_stats(x)` | `x: str` | `dict` | Returns a dictionary containing all extracted feature statistics at once. |
203
+ | `text_stats_batch(df, col)`| `df, col` | `DataFrame` | Returns a DataFrame of stats for a pandas DataFrame column. |
151
204
 
152
205
  ---
153
206
 
@@ -169,7 +222,19 @@ Below is the complete reference guide for all **34 functions** available in the
169
222
  | `remove_special_chars(x)` | `x: str` | `str` | Strips punctuation and special symbols. |
170
223
  | `remove_repeated_chars(x)` | `x: str` | `str` | Truncates repeated characters beyond 2 consecutive occurrences. |
171
224
  | `remove_stop_words(x)` | `x: str` | `str` | Case-insensitive removal of stop words while preserving word case. |
172
- | `clean_text(text)` | `text: str` | `str` | Runs end-to-end cleaning pipeline (lowercasing, contractions, emails, URLs, HTML, special chars, lemmatization). |
225
+ | `remove_emojis(x)` | `x: str` | `str` | Strips emoji characters from text. |
226
+ | `extract_emojis(x)` | `x: str` | `list[str]` | Returns list of emojis found in text. |
227
+ | `emoji_to_text(x)` | `x: str` | `str` | Converts emojis to descriptive text (e.g. 😂 -> `face_with_tears_of_joy`). |
228
+ | `remove_phone_numbers(x)` | `x: str` | `str` | Strips phone numbers from text. |
229
+ | `count_phone_numbers(x)` | `x: str` | `int` | Counts phone numbers present in text. |
230
+ | `mask_pii(x)` | `x: str` | `str` | Redacts emails, phone numbers, and card numbers with `[REDACTED]`. |
231
+ | `remove_extra_whitespace(x)`| `x: str` | `str` | Collapses consecutive spaces, tabs, and newlines to a single space. |
232
+ | `expand_hashtags(x)` | `x: str` | `str` | Splits camelCase/PascalCase hashtags (`#MachineLearning` -> `Machine Learning`). |
233
+ | `normalize_unicode_punctuation(x)`| `x: str` | `str` | Replaces smart quotes, em-dashes, and ellipsis with ASCII equivalents. |
234
+ | `remove_numbers(x)` | `x: str` | `str` | Strips all numeric digits. |
235
+ | `remove_currency_symbols(x)`| `x: str` | `str` | Strips currency symbols (`$`, `€`, `₹`, `£`, `¥`). |
236
+ | `normalize_whitespace_and_casing(x)`| `x: str` | `str` | Canonical combination of lowercasing and whitespace collapse. |
237
+ | `clean_text(text)` | `text: str` | `str` | Runs end-to-end cleaning pipeline. |
173
238
 
174
239
  ---
175
240
 
@@ -179,16 +244,38 @@ Below is the complete reference guide for all **34 functions** available in the
179
244
  | :--- | :--- | :--- | :--- |
180
245
  | `convert_to_base(x)` | `x: str` | `str` | Lemmatizes nouns and verbs via spaCy while keeping other POS tags. |
181
246
  | `lemmatize(x)` | `x: str` | `str` | Performs full lemmatization across all tokens via spaCy. |
247
+ | `extract_entities(x)` | `x: str` | `list[tuple]` | Extracts named entities `(text, label)` using spaCy. |
248
+ | `extract_pos_tags(x)` | `x: str` | `list[tuple]` | Extracts full POS tag breakdown `(token, pos, tag)` using spaCy. |
249
+ | `get_dependency_tree(x)`| `x: str` | `list[tuple]` | Extracts dependency parse tree `(token, dep, head)` using spaCy. |
250
+ | `detect_language(x)` | `x: str` | `str` | Detects language code (`'en'`, `'fr'`, etc.) using TextBlob. |
251
+ | `keyword_extraction(x, top_n)`| `x: str, top_n: int`| `list[str]`| Extracts top N keywords by term frequency. |
252
+ | `text_similarity(x1, x2)` | `x1, x2` | `float` | Calculates token Jaccard similarity between two texts (0.0 to 1.0). |
253
+ | `get_synonyms(word)` | `word: str` | `list[str]` | Returns list of synonyms from WordNet. |
254
+ | `get_antonyms(word)` | `word: str` | `list[str]` | Returns list of antonyms from WordNet. |
255
+ | `chunk_noun_phrases(x)` | `x: str` | `list[dict]` | Extracts noun phrases with character start/end spans. |
182
256
  | `correct_spelling(x)` | `x: str` | `str` | Corrects word spelling using TextBlob model. |
183
257
  | `get_noun_phrase(x)` | `x: str` | `list[str]` | Extracts noun phrases from text. |
184
258
  | `n_grams(x, n=2)` | `x: str, n: int` | `list` | Generates word-level n-grams. |
185
259
  | `singularize_words(x)` | `x: str` | `str` | Converts plural nouns (`NNS`) into singular forms. |
186
260
  | `pluralize_words(x)` | `x: str` | `str` | Converts singular nouns (`NN`) into plural forms. |
261
+
262
+ ---
263
+
264
+ ### 4. Safety, Moderation and Sentiment
265
+
266
+ | Function | Parameters | Return Type | Description |
267
+ | :--- | :--- | :--- | :--- |
268
+ | `detect_profanity(x)` | `x: str` | `bool` | Returns True if text contains profanity words. |
269
+ | `censor_profanity(x)` | `x: str` | `str` | Censors profanity words with asterisks (`****`). |
270
+ | `detect_toxicity(x)` | `x: str` | `float` | Calculates baseline toxicity score (0.0 to 1.0). |
271
+ | `detect_spam_signals(x)` | `x: str` | `dict` | Returns spam flags for excessive caps, punctuation, and URL density. |
187
272
  | `sentiment_analysis(x)` | `x: str` | `str` | Returns sentiment classification (`'pos'` / `'neg'`) using cached NaiveBayes model. |
273
+ | `polarity_subjectivity(x)`| `x: str` | `dict` | Returns fast pattern-based sentiment polarity (-1.0 to 1.0) and subjectivity. |
274
+ | `emotion_detection(x)` | `x: str` | `dict` | Calculates lexicon-based emotion counts (joy, anger, sadness, fear). |
188
275
 
189
276
  ---
190
277
 
191
- ### 4. Visualization Suite
278
+ ### 5. Visualization Suite
192
279
 
193
280
  | Function | Parameters | Return Type | Description |
194
281
  | :--- | :--- | :--- | :--- |
@@ -210,7 +297,20 @@ print(cleaned)
210
297
  # Output: check out contact support company org or i am love nlp
211
298
  ```
212
299
 
213
- ### 2. Pandas DataFrame Integration
300
+ ### 2. Pipeline Engine Usage
301
+
302
+ ```python
303
+ import nlp_text_preprocessing as tp
304
+
305
+ # Build custom pipeline
306
+ pipe = tp.Pipeline(['to_lower_case', 'expand_hashtags', 'remove_urls', 'remove_extra_whitespace'])
307
+
308
+ text = "Check out #MachineLearning at https://example.com!"
309
+ print(pipe.run(text))
310
+ # Output: check out machine learning at!
311
+ ```
312
+
313
+ ### 3. Pandas DataFrame Integration
214
314
 
215
315
  All functions seamlessly support Pandas `.apply()` and handle missing values (`NaN`/`None`):
216
316
 
@@ -233,7 +333,7 @@ df['clean_text'] = df['text'].apply(tp.clean_text)
233
333
  df['word_count'] = df['text'].apply(tp.word_count)
234
334
  ```
235
335
 
236
- ### 3. Batch Feature Extraction
336
+ ### 4. Batch Feature Extraction
237
337
 
238
338
  Extract full feature matrices directly into Pandas columns:
239
339
 
@@ -243,12 +343,12 @@ import nlp_text_preprocessing as tp
243
343
 
244
344
  df = pd.DataFrame({'text': ["Hello #world! Visit https://example.com", "Contact user@test.com"]})
245
345
 
246
- # Extract all features into a new DataFrame
247
- features_df = df['text'].apply(tp.extract_features).apply(pd.Series)
248
- print(features_df)
346
+ # Extract all summary stats into a new DataFrame
347
+ stats_df = tp.text_stats_batch(df, 'text')
348
+ print(stats_df)
249
349
  ```
250
350
 
251
- ### 4. WordCloud Generation
351
+ ### 5. WordCloud Generation
252
352
 
253
353
  ```python
254
354
  import nlp_text_preprocessing as tp
@@ -279,5 +379,5 @@ python -m spacy download en_core_web_sm
279
379
 
280
380
  ## License and Author
281
381
 
282
- - **Author**: [Uditya Narayan Tiwari](https://youtube.com/kgptalkie)
382
+ - **Author**: [Uditya Narayan Tiwari](https://udityanarayantiwari.netlify.app/)
283
383
  - **License**: MIT License ([LICENSE](LICENSE))