nlp-text-preprocessing 0.1.2__tar.gz → 0.2.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (20) hide show
  1. {nlp_text_preprocessing-0.1.2/nlp_text_preprocessing.egg-info → nlp_text_preprocessing-0.2.0}/PKG-INFO +113 -21
  2. {nlp_text_preprocessing-0.1.2 → nlp_text_preprocessing-0.2.0}/Readme.md +112 -20
  3. {nlp_text_preprocessing-0.1.2 → nlp_text_preprocessing-0.2.0}/nlp_text_preprocessing/__init__.py +38 -0
  4. nlp_text_preprocessing-0.2.0/nlp_text_preprocessing/all_text_preprocess.py +720 -0
  5. {nlp_text_preprocessing-0.1.2 → nlp_text_preprocessing-0.2.0/nlp_text_preprocessing.egg-info}/PKG-INFO +113 -21
  6. {nlp_text_preprocessing-0.1.2 → nlp_text_preprocessing-0.2.0}/setup.py +1 -1
  7. nlp_text_preprocessing-0.2.0/tests/test_all_text_preprocess.py +213 -0
  8. nlp_text_preprocessing-0.1.2/nlp_text_preprocessing/all_text_preprocess.py +0 -318
  9. nlp_text_preprocessing-0.1.2/tests/test_all_text_preprocess.py +0 -139
  10. {nlp_text_preprocessing-0.1.2 → nlp_text_preprocessing-0.2.0}/LICENSE +0 -0
  11. {nlp_text_preprocessing-0.1.2 → nlp_text_preprocessing-0.2.0}/MANIFEST.in +0 -0
  12. {nlp_text_preprocessing-0.1.2 → nlp_text_preprocessing-0.2.0}/architecture_diagram.png +0 -0
  13. {nlp_text_preprocessing-0.1.2 → nlp_text_preprocessing-0.2.0}/nlp_banner.png +0 -0
  14. {nlp_text_preprocessing-0.1.2 → nlp_text_preprocessing-0.2.0}/nlp_text_preprocessing/data/contractions.json +0 -0
  15. {nlp_text_preprocessing-0.1.2 → nlp_text_preprocessing-0.2.0}/nlp_text_preprocessing.egg-info/SOURCES.txt +0 -0
  16. {nlp_text_preprocessing-0.1.2 → nlp_text_preprocessing-0.2.0}/nlp_text_preprocessing.egg-info/dependency_links.txt +0 -0
  17. {nlp_text_preprocessing-0.1.2 → nlp_text_preprocessing-0.2.0}/nlp_text_preprocessing.egg-info/requires.txt +0 -0
  18. {nlp_text_preprocessing-0.1.2 → nlp_text_preprocessing-0.2.0}/nlp_text_preprocessing.egg-info/top_level.txt +0 -0
  19. {nlp_text_preprocessing-0.1.2 → nlp_text_preprocessing-0.2.0}/requirements.txt +0 -0
  20. {nlp_text_preprocessing-0.1.2 → nlp_text_preprocessing-0.2.0}/setup.cfg +0 -0
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: nlp_text_preprocessing
3
- Version: 0.1.2
3
+ Version: 0.2.0
4
4
  Summary: This is a Text Processing Package For NLP
5
5
  Author: Uditya Narayan Tiwari
6
6
  Author-email: tiwarimerit@gmail.com
@@ -66,16 +66,19 @@ Dynamic: summary
66
66
  - [Key Features](#key-features)
67
67
  - [Architecture Diagram](#architecture-diagram)
68
68
  - [Quickstart and Setup](#quickstart-and-setup)
69
+ - [Pipeline Automation Engine](#pipeline-automation-engine)
69
70
  - [Complete API Reference](#complete-api-reference)
70
71
  - [1. General Feature Extraction](#1-general-feature-extraction)
71
72
  - [2. Text Cleaning and Normalization](#2-text-cleaning-and-normalization)
72
73
  - [3. Linguistic and Morphological Processing](#3-linguistic-and-morphological-processing)
73
- - [4. Visualization Suite](#4-visualization-suite)
74
+ - [4. Safety, Moderation and Sentiment](#4-safety-moderation-and-sentiment)
75
+ - [5. Visualization Suite](#5-visualization-suite)
74
76
  - [Code Examples](#code-examples)
75
77
  - [Basic Text Cleaning](#1-basic-text-cleaning)
76
- - [Pandas DataFrame Integration](#2-pandas-dataframe-integration)
77
- - [Batch Feature Extraction](#3-batch-feature-extraction)
78
- - [WordCloud Generation](#4-wordcloud-generation)
78
+ - [Pipeline Engine Usage](#2-pipeline-engine-usage)
79
+ - [Pandas DataFrame Integration](#3-pandas-dataframe-integration)
80
+ - [Batch Feature Extraction](#4-batch-feature-extraction)
81
+ - [WordCloud Generation](#5-wordcloud-generation)
79
82
  - [Troubleshooting and Resource Setup](#troubleshooting-and-resource-setup)
80
83
  - [License and Author](#license-and-author)
81
84
 
@@ -83,20 +86,22 @@ Dynamic: summary
83
86
 
84
87
  ## Package Overview
85
88
 
86
- **`nlp_text_preprocessing`** is a production-grade, modular Python library designed to streamline text cleaning, feature extraction, linguistic analysis, and visualization for Natural Language Processing (NLP) pipelines and Machine Learning workflows.
89
+ **`nlp_text_preprocessing`** is a production-grade, modular Python library designed to streamline text cleaning, feature extraction, linguistic analysis, content moderation, and visualization for Natural Language Processing (NLP) pipelines and Machine Learning workflows.
87
90
 
88
- It provides **34 single-purpose functions** with built-in defensive guards for `None`, `NaN`, and missing values, making it 100% crash-proof when running over Pandas DataFrames or large raw text datasets.
91
+ It provides over **60 single-purpose functions** and an extensible **Pipeline Engine** with built-in defensive guards for `None`, `NaN`, and missing values, making it 100% crash-proof when running over Pandas DataFrames or large raw text datasets.
89
92
 
90
93
  ---
91
94
 
92
95
  ## Key Features
93
96
 
94
97
  - **Crash-Proof Pipeline**: Automatic `None`/`NaN` input sanitation across all functions.
98
+ - **Pipeline Automation Engine**: Chain cleaning steps by name or custom functions via `Pipeline(['to_lower_case', 'remove_urls', 'lemmatize'])`.
95
99
  - **Lazy and Fast Model Caching**: Efficient lazy-loading for spaCy (`en_core_web_sm`) and cached NaiveBayes sentiment analyzers.
96
- - **Comprehensive Cleaning**: Strip emails, URLs, HTML, retweets, special symbols, and expand contractions.
97
- - **Feature Extraction**: Extract word counts, character counts, uppercase counts, numerics, hashtags, mentions, and full feature matrices in one call.
98
- - **Linguistic Utilities**: Perform lemmatization, noun phrase extraction, spelling correction, n-grams, singularization, and pluralization.
99
- - **Flexible Visualizations**: Generate and save custom WordClouds to disk or display interactively.
100
+ - **Advanced Cleaning**: Remove emojis, phone numbers, PII mask, URLs, emails, HTML, retweets, special symbols, and expand contractions/hashtags.
101
+ - **Feature & Readability Extraction**: Extract word counts, sentence counts, readability scores (Flesch Ease), punctuation density, capital ratios, and full batch statistics.
102
+ - **Linguistic Utilities**: Named Entity Recognition (NER), POS tagging, dependency parsing, language detection, keyword extraction, synonyms, antonyms, and n-grams.
103
+ - **Safety & Moderation**: Profanity detection/censoring, toxicity scores, and spam signal detection.
104
+ - **Visualization**: Generate and save custom WordClouds to disk or display interactively.
100
105
 
101
106
  ---
102
107
 
@@ -131,9 +136,42 @@ tp.download_nltk_packages()
131
136
 
132
137
  ---
133
138
 
139
+ ## Pipeline Automation Engine
140
+
141
+ The `Pipeline` class allows chaining cleaning and transformation steps cleanly:
142
+
143
+ ```python
144
+ import nlp_text_preprocessing as tp
145
+
146
+ # 1. Create default zero-config pipeline
147
+ pipe = tp.Pipeline.default()
148
+
149
+ # 2. Or build a custom step pipeline
150
+ pipe = tp.Pipeline([
151
+ 'to_lower_case',
152
+ 'expand_hashtags',
153
+ 'remove_urls',
154
+ 'remove_emails',
155
+ 'remove_extra_whitespace',
156
+ 'lemmatize'
157
+ ])
158
+
159
+ # Process single text
160
+ clean_result = pipe.run("Check out #MachineLearning at https://example.com!")
161
+
162
+ # Process batch list or Pandas Series
163
+ clean_batch = pipe.run_batch(["Text 1", "Text 2"])
164
+ clean_series = pipe.run_series(df['text_column'])
165
+
166
+ # Generate diff report of changes
167
+ report = tp.diff_report(raw_texts, clean_batch)
168
+ ```
169
+
170
+ ---
171
+
134
172
  ## Complete API Reference
135
173
 
136
- Below is the complete reference guide for all **34 functions** available in the library:
174
+ Below is the complete reference guide for all functions available in the library:
137
175
 
138
176
  ### 1. General Feature Extraction
139
177
 
@@ -143,11 +181,18 @@ Below is the complete reference guide for all **34 functions** available in the
143
181
  | `char_count(x)` | `x: str` | `int` | Returns total characters excluding spaces. |
144
182
  | `avg_word_len(x)` | `x: str` | `float` | Returns average word length (`0.0` for empty inputs). |
145
183
  | `stop_words_count(x)` | `x: str` | `int` | Returns count of stop words (case-insensitive). |
184
+ | `sentence_count(x)` | `x: str` | `int` | Returns total number of sentences. |
185
+ | `avg_sentence_len(x)` | `x: str` | `float` | Returns average sentence length in words. |
186
+ | `punctuation_density(x)`| `x: str` | `float` | Returns proportion of punctuation characters in text. |
187
+ | `capital_ratio(x)` | `x: str` | `float` | Returns proportion of uppercase words relative to total words. |
188
+ | `unique_word_ratio(x)` | `x: str` | `float` | Returns lexical diversity ratio (unique words / total words). |
189
+ | `readability_score(x)` | `x: str` | `float` | Calculates Flesch Reading Ease score. |
146
190
  | `hashtags_count(x)` | `x: str` | `int` | Returns count of hashtag tokens (`#tag`). |
147
191
  | `mentions_count(x)` | `x: str` | `int` | Returns count of user handle mentions (`@user`). |
148
192
  | `numerics_count(x)` | `x: str` | `int` | Returns count of numeric tokens. |
149
193
  | `upper_case_count(x)` | `x: str` | `int` | Returns count of uppercase words. |
150
- | `extract_features(x)` | `x: str` | `dict` | Returns a dictionary containing all extracted feature counts at once. |
194
+ | `text_stats(x)` | `x: str` | `dict` | Returns a dictionary containing all extracted feature statistics at once. |
195
+ | `text_stats_batch(df, col)`| `df, col` | `DataFrame` | Returns a DataFrame of stats for a pandas DataFrame column. |
151
196
 
152
197
  ---
153
198
 
@@ -169,7 +214,19 @@ Below is the complete reference guide for all **34 functions** available in the
169
214
  | `remove_special_chars(x)` | `x: str` | `str` | Strips punctuation and special symbols. |
170
215
  | `remove_repeated_chars(x)` | `x: str` | `str` | Truncates repeated characters beyond 2 consecutive occurrences. |
171
216
  | `remove_stop_words(x)` | `x: str` | `str` | Case-insensitive removal of stop words while preserving word case. |
172
- | `clean_text(text)` | `text: str` | `str` | Runs end-to-end cleaning pipeline (lowercasing, contractions, emails, URLs, HTML, special chars, lemmatization). |
217
+ | `remove_emojis(x)` | `x: str` | `str` | Strips emoji characters from text. |
218
+ | `extract_emojis(x)` | `x: str` | `list[str]` | Returns list of emojis found in text. |
219
+ | `emoji_to_text(x)` | `x: str` | `str` | Converts emojis to descriptive text (e.g. 😂 -> `face_with_tears_of_joy`). |
220
+ | `remove_phone_numbers(x)` | `x: str` | `str` | Strips phone numbers from text. |
221
+ | `count_phone_numbers(x)` | `x: str` | `int` | Counts phone numbers present in text. |
222
+ | `mask_pii(x)` | `x: str` | `str` | Redacts emails, phone numbers, and card numbers with `[REDACTED]`. |
223
+ | `remove_extra_whitespace(x)`| `x: str` | `str` | Collapses consecutive spaces, tabs, and newlines to a single space. |
224
+ | `expand_hashtags(x)` | `x: str` | `str` | Splits camelCase/PascalCase hashtags (`#MachineLearning` -> `Machine Learning`). |
225
+ | `normalize_unicode_punctuation(x)`| `x: str` | `str` | Replaces smart quotes, em-dashes, and ellipsis with ASCII equivalents. |
226
+ | `remove_numbers(x)` | `x: str` | `str` | Strips all numeric digits. |
227
+ | `remove_currency_symbols(x)`| `x: str` | `str` | Strips currency symbols (`$`, `€`, `₹`, `£`, `¥`). |
228
+ | `normalize_whitespace_and_casing(x)`| `x: str` | `str` | Canonical combination of lowercasing and whitespace collapse. |
229
+ | `clean_text(text)` | `text: str` | `str` | Runs end-to-end cleaning pipeline. |
173
230
 
174
231
  ---
175
232
 
@@ -179,16 +236,38 @@ Below is the complete reference guide for all **34 functions** available in the
179
236
  | :--- | :--- | :--- | :--- |
180
237
  | `convert_to_base(x)` | `x: str` | `str` | Lemmatizes nouns and verbs via spaCy while keeping other POS tags. |
181
238
  | `lemmatize(x)` | `x: str` | `str` | Performs full lemmatization across all tokens via spaCy. |
239
+ | `extract_entities(x)` | `x: str` | `list[tuple]` | Extracts named entities `(text, label)` using spaCy. |
240
+ | `extract_pos_tags(x)` | `x: str` | `list[tuple]` | Extracts full POS tag breakdown `(token, pos, tag)` using spaCy. |
241
+ | `get_dependency_tree(x)`| `x: str` | `list[tuple]` | Extracts dependency parse tree `(token, dep, head)` using spaCy. |
242
+ | `detect_language(x)` | `x: str` | `str` | Detects language code (`'en'`, `'fr'`, etc.) using TextBlob. |
243
+ | `keyword_extraction(x, top_n)`| `x: str, top_n: int`| `list[str]`| Extracts top N keywords by term frequency. |
244
+ | `text_similarity(x1, x2)` | `x1, x2` | `float` | Calculates token Jaccard similarity between two texts (0.0 to 1.0). |
245
+ | `get_synonyms(word)` | `word: str` | `list[str]` | Returns list of synonyms from WordNet. |
246
+ | `get_antonyms(word)` | `word: str` | `list[str]` | Returns list of antonyms from WordNet. |
247
+ | `chunk_noun_phrases(x)` | `x: str` | `list[dict]` | Extracts noun phrases with character start/end spans. |
182
248
  | `correct_spelling(x)` | `x: str` | `str` | Corrects word spelling using TextBlob model. |
183
249
  | `get_noun_phrase(x)` | `x: str` | `list[str]` | Extracts noun phrases from text. |
184
250
  | `n_grams(x, n=2)` | `x: str, n: int` | `list` | Generates word-level n-grams. |
185
251
  | `singularize_words(x)` | `x: str` | `str` | Converts plural nouns (`NNS`) into singular forms. |
186
252
  | `pluralize_words(x)` | `x: str` | `str` | Converts singular nouns (`NN`) into plural forms. |
253
+
254
+ ---
255
+
256
+ ### 4. Safety, Moderation and Sentiment
257
+
258
+ | Function | Parameters | Return Type | Description |
259
+ | :--- | :--- | :--- | :--- |
260
+ | `detect_profanity(x)` | `x: str` | `bool` | Returns True if text contains profanity words. |
261
+ | `censor_profanity(x)` | `x: str` | `str` | Censors profanity words with asterisks (`****`). |
262
+ | `detect_toxicity(x)` | `x: str` | `float` | Calculates baseline toxicity score (0.0 to 1.0). |
263
+ | `detect_spam_signals(x)` | `x: str` | `dict` | Returns spam flags for excessive caps, punctuation, and URL density. |
187
264
  | `sentiment_analysis(x)` | `x: str` | `str` | Returns sentiment classification (`'pos'` / `'neg'`) using cached NaiveBayes model. |
265
+ | `polarity_subjectivity(x)`| `x: str` | `dict` | Returns fast pattern-based sentiment polarity (-1.0 to 1.0) and subjectivity. |
266
+ | `emotion_detection(x)` | `x: str` | `dict` | Calculates lexicon-based emotion counts (joy, anger, sadness, fear). |
188
267
 
189
268
  ---
190
269
 
191
- ### 4. Visualization Suite
270
+ ### 5. Visualization Suite
192
271
 
193
272
  | Function | Parameters | Return Type | Description |
194
273
  | :--- | :--- | :--- | :--- |
@@ -210,7 +289,20 @@ print(cleaned)
210
289
  # Output: check out contact support company org or i am love nlp
211
290
  ```
212
291
 
213
- ### 2. Pandas DataFrame Integration
292
+ ### 2. Pipeline Engine Usage
293
+
294
+ ```python
295
+ import nlp_text_preprocessing as tp
296
+
297
+ # Build custom pipeline
298
+ pipe = tp.Pipeline(['to_lower_case', 'expand_hashtags', 'remove_urls', 'remove_extra_whitespace'])
299
+
300
+ text = "Check out #MachineLearning at https://example.com!"
301
+ print(pipe.run(text))
302
+ # Output: check out machine learning at!
303
+ ```
304
+
305
+ ### 3. Pandas DataFrame Integration
214
306
 
215
307
  All functions seamlessly support Pandas `.apply()` and handle missing values (`NaN`/`None`):
216
308
 
@@ -233,7 +325,7 @@ df['clean_text'] = df['text'].apply(tp.clean_text)
233
325
  df['word_count'] = df['text'].apply(tp.word_count)
234
326
  ```
235
327
 
236
- ### 3. Batch Feature Extraction
328
+ ### 4. Batch Feature Extraction
237
329
 
238
330
  Extract full feature matrices directly into Pandas columns:
239
331
 
@@ -243,12 +335,12 @@ import nlp_text_preprocessing as tp
243
335
 
244
336
  df = pd.DataFrame({'text': ["Hello #world! Visit https://example.com", "Contact user@test.com"]})
245
337
 
246
- # Extract all features into a new DataFrame
247
- features_df = df['text'].apply(tp.extract_features).apply(pd.Series)
248
- print(features_df)
338
+ # Extract all summary stats into a new DataFrame
339
+ stats_df = tp.text_stats_batch(df, 'text')
340
+ print(stats_df)
249
341
  ```
250
342
 
251
- ### 4. WordCloud Generation
343
+ ### 5. WordCloud Generation
252
344
 
253
345
  ```python
254
346
  import nlp_text_preprocessing as tp
@@ -30,16 +30,19 @@
30
30
  - [Key Features](#key-features)
31
31
  - [Architecture Diagram](#architecture-diagram)
32
32
  - [Quickstart and Setup](#quickstart-and-setup)
33
+ - [Pipeline Automation Engine](#pipeline-automation-engine)
33
34
  - [Complete API Reference](#complete-api-reference)
34
35
  - [1. General Feature Extraction](#1-general-feature-extraction)
35
36
  - [2. Text Cleaning and Normalization](#2-text-cleaning-and-normalization)
36
37
  - [3. Linguistic and Morphological Processing](#3-linguistic-and-morphological-processing)
37
- - [4. Visualization Suite](#4-visualization-suite)
38
+ - [4. Safety, Moderation and Sentiment](#4-safety-moderation-and-sentiment)
39
+ - [5. Visualization Suite](#5-visualization-suite)
38
40
  - [Code Examples](#code-examples)
39
41
  - [Basic Text Cleaning](#1-basic-text-cleaning)
40
- - [Pandas DataFrame Integration](#2-pandas-dataframe-integration)
41
- - [Batch Feature Extraction](#3-batch-feature-extraction)
42
- - [WordCloud Generation](#4-wordcloud-generation)
42
+ - [Pipeline Engine Usage](#2-pipeline-engine-usage)
43
+ - [Pandas DataFrame Integration](#3-pandas-dataframe-integration)
44
+ - [Batch Feature Extraction](#4-batch-feature-extraction)
45
+ - [WordCloud Generation](#5-wordcloud-generation)
43
46
  - [Troubleshooting and Resource Setup](#troubleshooting-and-resource-setup)
44
47
  - [License and Author](#license-and-author)
45
48
 
@@ -47,20 +50,22 @@
47
50
 
48
51
  ## Package Overview
49
52
 
50
- **`nlp_text_preprocessing`** is a production-grade, modular Python library designed to streamline text cleaning, feature extraction, linguistic analysis, and visualization for Natural Language Processing (NLP) pipelines and Machine Learning workflows.
53
+ **`nlp_text_preprocessing`** is a production-grade, modular Python library designed to streamline text cleaning, feature extraction, linguistic analysis, content moderation, and visualization for Natural Language Processing (NLP) pipelines and Machine Learning workflows.
51
54
 
52
- It provides **34 single-purpose functions** with built-in defensive guards for `None`, `NaN`, and missing values, making it 100% crash-proof when running over Pandas DataFrames or large raw text datasets.
55
+ It provides over **60 single-purpose functions** and an extensible **Pipeline Engine** with built-in defensive guards for `None`, `NaN`, and missing values, making it 100% crash-proof when running over Pandas DataFrames or large raw text datasets.
53
56
 
54
57
  ---
55
58
 
56
59
  ## Key Features
57
60
 
58
61
  - **Crash-Proof Pipeline**: Automatic `None`/`NaN` input sanitation across all functions.
62
+ - **Pipeline Automation Engine**: Chain cleaning steps by name or custom functions via `Pipeline(['to_lower_case', 'remove_urls', 'lemmatize'])`.
59
63
  - **Lazy and Fast Model Caching**: Efficient lazy-loading for spaCy (`en_core_web_sm`) and cached NaiveBayes sentiment analyzers.
60
- - **Comprehensive Cleaning**: Strip emails, URLs, HTML, retweets, special symbols, and expand contractions.
61
- - **Feature Extraction**: Extract word counts, character counts, uppercase counts, numerics, hashtags, mentions, and full feature matrices in one call.
62
- - **Linguistic Utilities**: Perform lemmatization, noun phrase extraction, spelling correction, n-grams, singularization, and pluralization.
63
- - **Flexible Visualizations**: Generate and save custom WordClouds to disk or display interactively.
64
+ - **Advanced Cleaning**: Remove emojis, phone numbers, PII mask, URLs, emails, HTML, retweets, special symbols, and expand contractions/hashtags.
65
+ - **Feature & Readability Extraction**: Extract word counts, sentence counts, readability scores (Flesch Ease), punctuation density, capital ratios, and full batch statistics.
66
+ - **Linguistic Utilities**: Named Entity Recognition (NER), POS tagging, dependency parsing, language detection, keyword extraction, synonyms, antonyms, and n-grams.
67
+ - **Safety & Moderation**: Profanity detection/censoring, toxicity scores, and spam signal detection.
68
+ - **Visualization**: Generate and save custom WordClouds to disk or display interactively.
64
69
 
65
70
  ---
66
71
 
@@ -95,9 +100,42 @@ tp.download_nltk_packages()
95
100
 
96
101
  ---
97
102
 
103
+ ## Pipeline Automation Engine
104
+
105
+ The `Pipeline` class allows chaining cleaning and transformation steps cleanly:
106
+
107
+ ```python
108
+ import nlp_text_preprocessing as tp
109
+
110
+ # 1. Create default zero-config pipeline
111
+ pipe = tp.Pipeline.default()
112
+
113
+ # 2. Or build a custom step pipeline
114
+ pipe = tp.Pipeline([
115
+ 'to_lower_case',
116
+ 'expand_hashtags',
117
+ 'remove_urls',
118
+ 'remove_emails',
119
+ 'remove_extra_whitespace',
120
+ 'lemmatize'
121
+ ])
122
+
123
+ # Process single text
124
+ clean_result = pipe.run("Check out #MachineLearning at https://example.com!")
125
+
126
+ # Process batch list or Pandas Series
127
+ clean_batch = pipe.run_batch(["Text 1", "Text 2"])
128
+ clean_series = pipe.run_series(df['text_column'])
129
+
130
+ # Generate diff report of changes
131
+ report = tp.diff_report(raw_texts, clean_batch)
132
+ ```
133
+
134
+ ---
135
+
98
136
  ## Complete API Reference
99
137
 
100
- Below is the complete reference guide for all **34 functions** available in the library:
138
+ Below is the complete reference guide for all functions available in the library:
101
139
 
102
140
  ### 1. General Feature Extraction
103
141
 
@@ -107,11 +145,18 @@ Below is the complete reference guide for all **34 functions** available in the
107
145
  | `char_count(x)` | `x: str` | `int` | Returns total characters excluding spaces. |
108
146
  | `avg_word_len(x)` | `x: str` | `float` | Returns average word length (`0.0` for empty inputs). |
109
147
  | `stop_words_count(x)` | `x: str` | `int` | Returns count of stop words (case-insensitive). |
148
+ | `sentence_count(x)` | `x: str` | `int` | Returns total number of sentences. |
149
+ | `avg_sentence_len(x)` | `x: str` | `float` | Returns average sentence length in words. |
150
+ | `punctuation_density(x)`| `x: str` | `float` | Returns proportion of punctuation characters in text. |
151
+ | `capital_ratio(x)` | `x: str` | `float` | Returns proportion of uppercase words relative to total words. |
152
+ | `unique_word_ratio(x)` | `x: str` | `float` | Returns lexical diversity ratio (unique words / total words). |
153
+ | `readability_score(x)` | `x: str` | `float` | Calculates Flesch Reading Ease score. |
110
154
  | `hashtags_count(x)` | `x: str` | `int` | Returns count of hashtag tokens (`#tag`). |
111
155
  | `mentions_count(x)` | `x: str` | `int` | Returns count of user handle mentions (`@user`). |
112
156
  | `numerics_count(x)` | `x: str` | `int` | Returns count of numeric tokens. |
113
157
  | `upper_case_count(x)` | `x: str` | `int` | Returns count of uppercase words. |
114
- | `extract_features(x)` | `x: str` | `dict` | Returns a dictionary containing all extracted feature counts at once. |
158
+ | `text_stats(x)` | `x: str` | `dict` | Returns a dictionary containing all extracted feature statistics at once. |
159
+ | `text_stats_batch(df, col)`| `df, col` | `DataFrame` | Returns a DataFrame of stats for a pandas DataFrame column. |
115
160
 
116
161
  ---
117
162
 
@@ -133,7 +178,19 @@ Below is the complete reference guide for all **34 functions** available in the
133
178
  | `remove_special_chars(x)` | `x: str` | `str` | Strips punctuation and special symbols. |
134
179
  | `remove_repeated_chars(x)` | `x: str` | `str` | Truncates repeated characters beyond 2 consecutive occurrences. |
135
180
  | `remove_stop_words(x)` | `x: str` | `str` | Case-insensitive removal of stop words while preserving word case. |
136
- | `clean_text(text)` | `text: str` | `str` | Runs end-to-end cleaning pipeline (lowercasing, contractions, emails, URLs, HTML, special chars, lemmatization). |
181
+ | `remove_emojis(x)` | `x: str` | `str` | Strips emoji characters from text. |
182
+ | `extract_emojis(x)` | `x: str` | `list[str]` | Returns list of emojis found in text. |
183
+ | `emoji_to_text(x)` | `x: str` | `str` | Converts emojis to descriptive text (e.g. 😂 -> `face_with_tears_of_joy`). |
184
+ | `remove_phone_numbers(x)` | `x: str` | `str` | Strips phone numbers from text. |
185
+ | `count_phone_numbers(x)` | `x: str` | `int` | Counts phone numbers present in text. |
186
+ | `mask_pii(x)` | `x: str` | `str` | Redacts emails, phone numbers, and card numbers with `[REDACTED]`. |
187
+ | `remove_extra_whitespace(x)`| `x: str` | `str` | Collapses consecutive spaces, tabs, and newlines to a single space. |
188
+ | `expand_hashtags(x)` | `x: str` | `str` | Splits camelCase/PascalCase hashtags (`#MachineLearning` -> `Machine Learning`). |
189
+ | `normalize_unicode_punctuation(x)`| `x: str` | `str` | Replaces smart quotes, em-dashes, and ellipsis with ASCII equivalents. |
190
+ | `remove_numbers(x)` | `x: str` | `str` | Strips all numeric digits. |
191
+ | `remove_currency_symbols(x)`| `x: str` | `str` | Strips currency symbols (`$`, `€`, `₹`, `£`, `¥`). |
192
+ | `normalize_whitespace_and_casing(x)`| `x: str` | `str` | Canonical combination of lowercasing and whitespace collapse. |
193
+ | `clean_text(text)` | `text: str` | `str` | Runs end-to-end cleaning pipeline. |
137
194
 
138
195
  ---
139
196
 
@@ -143,16 +200,38 @@ Below is the complete reference guide for all **34 functions** available in the
143
200
  | :--- | :--- | :--- | :--- |
144
201
  | `convert_to_base(x)` | `x: str` | `str` | Lemmatizes nouns and verbs via spaCy while keeping other POS tags. |
145
202
  | `lemmatize(x)` | `x: str` | `str` | Performs full lemmatization across all tokens via spaCy. |
203
+ | `extract_entities(x)` | `x: str` | `list[tuple]` | Extracts named entities `(text, label)` using spaCy. |
204
+ | `extract_pos_tags(x)` | `x: str` | `list[tuple]` | Extracts full POS tag breakdown `(token, pos, tag)` using spaCy. |
205
+ | `get_dependency_tree(x)`| `x: str` | `list[tuple]` | Extracts dependency parse tree `(token, dep, head)` using spaCy. |
206
+ | `detect_language(x)` | `x: str` | `str` | Detects language code (`'en'`, `'fr'`, etc.) using TextBlob. |
207
+ | `keyword_extraction(x, top_n)`| `x: str, top_n: int`| `list[str]`| Extracts top N keywords by term frequency. |
208
+ | `text_similarity(x1, x2)` | `x1, x2` | `float` | Calculates token Jaccard similarity between two texts (0.0 to 1.0). |
209
+ | `get_synonyms(word)` | `word: str` | `list[str]` | Returns list of synonyms from WordNet. |
210
+ | `get_antonyms(word)` | `word: str` | `list[str]` | Returns list of antonyms from WordNet. |
211
+ | `chunk_noun_phrases(x)` | `x: str` | `list[dict]` | Extracts noun phrases with character start/end spans. |
146
212
  | `correct_spelling(x)` | `x: str` | `str` | Corrects word spelling using TextBlob model. |
147
213
  | `get_noun_phrase(x)` | `x: str` | `list[str]` | Extracts noun phrases from text. |
148
214
  | `n_grams(x, n=2)` | `x: str, n: int` | `list` | Generates word-level n-grams. |
149
215
  | `singularize_words(x)` | `x: str` | `str` | Converts plural nouns (`NNS`) into singular forms. |
150
216
  | `pluralize_words(x)` | `x: str` | `str` | Converts singular nouns (`NN`) into plural forms. |
217
+
218
+ ---
219
+
220
+ ### 4. Safety, Moderation and Sentiment
221
+
222
+ | Function | Parameters | Return Type | Description |
223
+ | :--- | :--- | :--- | :--- |
224
+ | `detect_profanity(x)` | `x: str` | `bool` | Returns True if text contains profanity words. |
225
+ | `censor_profanity(x)` | `x: str` | `str` | Censors profanity words with asterisks (`****`). |
226
+ | `detect_toxicity(x)` | `x: str` | `float` | Calculates baseline toxicity score (0.0 to 1.0). |
227
+ | `detect_spam_signals(x)` | `x: str` | `dict` | Returns spam flags for excessive caps, punctuation, and URL density. |
151
228
  | `sentiment_analysis(x)` | `x: str` | `str` | Returns sentiment classification (`'pos'` / `'neg'`) using cached NaiveBayes model. |
229
+ | `polarity_subjectivity(x)`| `x: str` | `dict` | Returns fast pattern-based sentiment polarity (-1.0 to 1.0) and subjectivity. |
230
+ | `emotion_detection(x)` | `x: str` | `dict` | Calculates lexicon-based emotion counts (joy, anger, sadness, fear). |
152
231
 
153
232
  ---
154
233
 
155
- ### 4. Visualization Suite
234
+ ### 5. Visualization Suite
156
235
 
157
236
  | Function | Parameters | Return Type | Description |
158
237
  | :--- | :--- | :--- | :--- |
@@ -174,7 +253,20 @@ print(cleaned)
174
253
  # Output: check out contact support company org or i am love nlp
175
254
  ```
176
255
 
177
- ### 2. Pandas DataFrame Integration
256
+ ### 2. Pipeline Engine Usage
257
+
258
+ ```python
259
+ import nlp_text_preprocessing as tp
260
+
261
+ # Build custom pipeline
262
+ pipe = tp.Pipeline(['to_lower_case', 'expand_hashtags', 'remove_urls', 'remove_extra_whitespace'])
263
+
264
+ text = "Check out #MachineLearning at https://example.com!"
265
+ print(pipe.run(text))
266
+ # Output: check out machine learning at!
267
+ ```
268
+
269
+ ### 3. Pandas DataFrame Integration
178
270
 
179
271
  All functions seamlessly support Pandas `.apply()` and handle missing values (`NaN`/`None`):
180
272
 
@@ -197,7 +289,7 @@ df['clean_text'] = df['text'].apply(tp.clean_text)
197
289
  df['word_count'] = df['text'].apply(tp.word_count)
198
290
  ```
199
291
 
200
- ### 3. Batch Feature Extraction
292
+ ### 4. Batch Feature Extraction
201
293
 
202
294
  Extract full feature matrices directly into Pandas columns:
203
295
 
@@ -207,12 +299,12 @@ import nlp_text_preprocessing as tp
207
299
 
208
300
  df = pd.DataFrame({'text': ["Hello #world! Visit https://example.com", "Contact user@test.com"]})
209
301
 
210
- # Extract all features into a new DataFrame
211
- features_df = df['text'].apply(tp.extract_features).apply(pd.Series)
212
- print(features_df)
302
+ # Extract all summary stats into a new DataFrame
303
+ stats_df = tp.text_stats_batch(df, 'text')
304
+ print(stats_df)
213
305
  ```
214
306
 
215
- ### 4. WordCloud Generation
307
+ ### 5. WordCloud Generation
216
308
 
217
309
  ```python
218
310
  import nlp_text_preprocessing as tp
@@ -12,6 +12,14 @@ __all__ = [
12
12
  'mentions_count',
13
13
  'numerics_count',
14
14
  'upper_case_count',
15
+ 'sentence_count',
16
+ 'avg_sentence_len',
17
+ 'punctuation_density',
18
+ 'capital_ratio',
19
+ 'unique_word_ratio',
20
+ 'readability_score',
21
+ 'text_stats',
22
+ 'text_stats_batch',
15
23
  'to_lower_case',
16
24
  'contraction_to_expansion',
17
25
  'remove_emails',
@@ -26,15 +34,45 @@ __all__ = [
26
34
  'remove_special_chars',
27
35
  'remove_repeated_chars',
28
36
  'remove_stop_words',
37
+ 'remove_emojis',
38
+ 'extract_emojis',
39
+ 'emoji_to_text',
40
+ 'remove_phone_numbers',
41
+ 'count_phone_numbers',
42
+ 'mask_pii',
43
+ 'remove_extra_whitespace',
44
+ 'expand_hashtags',
45
+ 'normalize_unicode_punctuation',
46
+ 'remove_numbers',
47
+ 'remove_currency_symbols',
48
+ 'normalize_whitespace_and_casing',
29
49
  'convert_to_base',
30
50
  'lemmatize',
51
+ 'extract_entities',
52
+ 'extract_pos_tags',
53
+ 'get_dependency_tree',
54
+ 'detect_language',
55
+ 'keyword_extraction',
56
+ 'text_similarity',
57
+ 'get_synonyms',
58
+ 'get_antonyms',
59
+ 'chunk_noun_phrases',
31
60
  'get_wordcloud',
32
61
  'correct_spelling',
33
62
  'get_noun_phrase',
34
63
  'n_grams',
35
64
  'singularize_words',
36
65
  'pluralize_words',
66
+ 'detect_profanity',
67
+ 'censor_profanity',
68
+ 'detect_toxicity',
69
+ 'detect_spam_signals',
37
70
  'sentiment_analysis',
71
+ 'polarity_subjectivity',
72
+ 'emotion_detection',
73
+ 'Pipeline',
74
+ 'register_step',
75
+ 'diff_report',
38
76
  'extract_features',
39
77
  'clean_text',
40
78
  ]