valx 0.2.4__tar.gz → 0.2.5__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,6 +1,6 @@
1
- Metadata-Version: 2.1
1
+ Metadata-Version: 2.4
2
2
  Name: valx
3
- Version: 0.2.4
3
+ Version: 0.2.5
4
4
  Summary: An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.
5
5
  Home-page: https://github.com/infinitode/valx
6
6
  Author: Infinitode Pty Ltd
@@ -19,6 +19,16 @@ Requires-Python: >=3.6
19
19
  Description-Content-Type: text/markdown
20
20
  License-File: LICENSE
21
21
  Requires-Dist: scikit-learn
22
+ Dynamic: author
23
+ Dynamic: author-email
24
+ Dynamic: classifier
25
+ Dynamic: description
26
+ Dynamic: description-content-type
27
+ Dynamic: home-page
28
+ Dynamic: license-file
29
+ Dynamic: requires-dist
30
+ Dynamic: requires-python
31
+ Dynamic: summary
22
32
 
23
33
  # ValX
24
34
  ![Python Version](https://img.shields.io/badge/python-3.12-blue.svg)
@@ -30,9 +40,29 @@ Requires-Dist: scikit-learn
30
40
  An open-source Python library for data cleaning tasks. It includes functions for profanity detection, and removal, and detection and removal of personal information. Also includes hate speech and offensive language detection and removal, using AI.
31
41
 
32
42
  > [!IMPORTANT]
43
+ > If you are using `scikit-learn` versions older than version `1.3.0`, please also downgrade your version of `numpy` as stated below. Otherwise, you can continue to use your preferred version of `scikit-learn` without downgrading `numpy`.
44
+ >
33
45
  > Please downgrade to `numpy` version `1.26.4`. Our ValX **DecisionTreeClassifier** AI model, relies on lower versions of `numpy`, because it was trained on these versions.
34
46
  > For more information see: https://techoverflow.net/2024/07/23/how-to-fix-numpy-dtype-size-changed-may-indicate-binary-incompatibility-expected-96-from-c-header-got-88-from-pyobject/
35
47
 
48
+ > [!NOTE]
49
+ > ValX will automatically install a version of `scikit-learn` that is compatible with your device if you don't have one already.
50
+
51
+ ## Changes in 0.2.5
52
+
53
+ ValX v0.2.5 introduces enhanced flexibility for profanity filtering by adding support for custom profanity lists:
54
+
55
+ - **Custom Profanity Word Lists**: Users can now provide their own lists of profane words directly as Python lists to the `detect_profanity` and `remove_profanity` functions via the new `custom_words_list` parameter.
56
+ - **Standalone Custom Lists**: Utilize your custom profanity list exclusively by setting the `language` parameter to `None`. ValX will then only use the words provided in `custom_words_list`.
57
+ - **Combined Lists**: Use a custom list in conjunction with ValX's built-in language-specific wordlists. Simply provide both a `language` (e.g., "English") and your `custom_words_list`. ValX will use the combined set of words.
58
+ - **Loading Custom Lists from File**: A new helper function, `load_custom_profanity_from_file(filepath)`, allows you to easily load custom profanity words from a text file.
59
+ - **File Format**: The file should contain one profanity word per line.
60
+ - Lines starting with a hash symbol (`#`) are treated as comments and ignored.
61
+ - Empty lines or lines containing only whitespace are also ignored.
62
+ - **Updated Detection Reporting**: The `detect_profanity` function's output now specifies the source of detected profanity more clearly (e.g., "Custom", "Custom + English").
63
+
64
+ These features give users greater control over the profanity filtering process, allowing for more tailored and specific use cases.
65
+
36
66
  ## Changes in 0.2.4
37
67
 
38
68
  Fixed a major incompatibility issue with `scikit-learn` due to version changes in `scikit-learn v1.3.0` which causes compatibility issues with versions later than `1.2.2`. ValX can now be used with `scikit-learn` versions earlier and later than `1.3.0`!
@@ -130,25 +160,129 @@ Below is a complete list of all the available supported languages for ValX's pro
130
160
 
131
161
  ## Usage
132
162
 
133
- ### Detect Profanity
163
+ ### Profanity Detection and Removal
164
+
165
+ ValX allows for flexible profanity filtering using built-in language lists, custom word lists (provided as Python lists or loaded from files), or a combination of both.
166
+
167
+ **1. Basic Profanity Detection (Built-in Language)**
134
168
 
135
169
  ```python
136
170
  from valx import detect_profanity
137
171
 
138
- # Detect profanity
172
+ sample_text = ["This is some fuck and porn text."]
173
+ # Detect profanity using the English list
139
174
  results = detect_profanity(sample_text, language='English')
140
- print("Profanity Evaluation Results", results)
175
+ # results will be:
176
+ # [
177
+ # {'Line': 1, 'Column': 14, 'Word': 'fuck', 'Language': 'English'},
178
+ # {'Line': 1, 'Column': 23, 'Word': 'porn', 'Language': 'English'}
179
+ # ]
180
+ print(results)
181
+ ```
182
+
183
+ **2. Profanity Detection with a Custom Word List (Python List)**
184
+
185
+ You can provide your own list of words to filter.
186
+
187
+ ```python
188
+ from valx import detect_profanity
189
+
190
+ sample_text = ["This contains custombadword1 and also asshole from English list."]
191
+ my_custom_words = ["custombadword1", "anothercustom"]
192
+
193
+ # Option A: Custom list ONLY (language=None)
194
+ results_custom_only = detect_profanity(sample_text, language=None, custom_words_list=my_custom_words)
195
+ # results_custom_only will detect "custombadword1" with Language: "Custom"
196
+ # [{'Line': 1, 'Column': 15, 'Word': 'custombadword1', 'Language': 'Custom'}]
197
+ print(results_custom_only)
198
+
199
+ # Option B: Custom list COMBINED with a built-in language
200
+ results_custom_plus_english = detect_profanity(sample_text, language="English", custom_words_list=my_custom_words)
201
+ # results_custom_plus_english will detect "custombadword1" and "asshole"
202
+ # Language will be "Custom + English"
203
+ # [
204
+ # {'Line': 1, 'Column': 15, 'Word': 'custombadword1', 'Language': 'Custom + English'},
205
+ # {'Line': 1, 'Column': 43, 'Word': 'asshole', 'Language': 'Custom + English'}
206
+ # ]
207
+ print(results_custom_plus_english)
141
208
  ```
142
209
 
143
- ### Remove Profanity
210
+ **3. Loading Custom Profanity Words from a File**
211
+
212
+ ValX provides a helper function to load words from a text file (one word per line, '#' for comments).
144
213
 
145
214
  ```python
146
- from valx import remove_profanity
215
+ from valx import detect_profanity, load_custom_profanity_from_file
216
+
217
+ # Assume 'my_profanity_file.txt' contains:
218
+ # customfileword1
219
+ # # this is a comment
220
+ # customfileword2
221
+
222
+ custom_words_from_file = load_custom_profanity_from_file("my_profanity_file.txt")
223
+ # custom_words_from_file will be: ['customfileword1', 'customfileword2']
224
+
225
+ sample_text_for_file = ["Text with customfileword1 and built-in shit."]
226
+
227
+ # Use file-loaded list with English built-in list
228
+ results_file_plus_english = detect_profanity(
229
+ sample_text_for_file,
230
+ language="English",
231
+ custom_words_list=custom_words_from_file
232
+ )
233
+ # Detects "customfileword1" and "shit", Language: "Custom + English"
234
+ print(results_file_plus_english)
235
+
236
+ # Use file-loaded list ONLY
237
+ results_file_only = detect_profanity(
238
+ sample_text_for_file,
239
+ language=None, # Important: set language to None
240
+ custom_words_list=custom_words_from_file
241
+ )
242
+ # Detects only "customfileword1", Language: "Custom"
243
+ print(results_file_only)
244
+ ```
245
+
246
+ **Output Format for `detect_profanity`**
147
247
 
148
- # Remove profanity
149
- removed = remove_profanity(sample_text, "text_cleaned.txt", language="English")
248
+ The `detect_profanity` function returns a list of dictionaries. Each dictionary includes:
249
+ - `"Line"`: The line number (1-indexed).
250
+ - `"Column"`: The column number (1-indexed) where the profanity starts.
251
+ - `"Word"`: The detected profanity word.
252
+ - `"Language"`: Indicates the source of the word list:
253
+ - `<LanguageName>` (e.g., "English"): If only a built-in language list was used.
254
+ - `"Custom"`: If `language=None` and only a `custom_words_list` was used.
255
+ - `"Custom + <LanguageName>"` (e.g., "Custom + English"): If both a built-in list and `custom_words_list` were used.
256
+ - `"Custom + All"`: If `language='All'` and `custom_words_list` were used.
257
+
258
+
259
+ **4. Removing Profanity**
260
+
261
+ `remove_profanity` works similarly, accepting `language` and `custom_words_list` parameters.
262
+
263
+ ```python
264
+ from valx import remove_profanity, load_custom_profanity_from_file
265
+
266
+ sample_text = ["This is fuck, custombadword1, and text with customfileword1."]
267
+ my_custom_words = ["custombadword1"]
268
+ custom_words_from_file = load_custom_profanity_from_file("my_profanity_file.txt") # Assuming it contains 'customfileword1'
269
+
270
+ # Remove profanity using English built-in + my_custom_words + custom_words_from_file
271
+ all_custom_words = list(set(my_custom_words + custom_words_from_file)) # Combine and unique
272
+
273
+ cleaned_text = remove_profanity(
274
+ sample_text,
275
+ output_file="cleaned_output.txt", # Optional: saves to file
276
+ language="English",
277
+ custom_words_list=all_custom_words
278
+ )
279
+ # cleaned_text will have "fuck", "custombadword1", and "customfileword1" replaced with "bad word".
280
+ # e.g., ["This is bad word, bad word, and text with bad word."]
281
+ print(cleaned_text)
150
282
  ```
151
283
 
284
+ The `load_profanity_words` function (used internally) also accepts `language` and `custom_words_list` if you need direct access to the word lists.
285
+
152
286
  ### Detect Sensitive Information
153
287
 
154
288
  ```python
@@ -8,9 +8,29 @@
8
8
  An open-source Python library for data cleaning tasks. It includes functions for profanity detection, and removal, and detection and removal of personal information. Also includes hate speech and offensive language detection and removal, using AI.
9
9
 
10
10
  > [!IMPORTANT]
11
+ > If you are using `scikit-learn` versions older than version `1.3.0`, please also downgrade your version of `numpy` as stated below. Otherwise, you can continue to use your preferred version of `scikit-learn` without downgrading `numpy`.
12
+ >
11
13
  > Please downgrade to `numpy` version `1.26.4`. Our ValX **DecisionTreeClassifier** AI model, relies on lower versions of `numpy`, because it was trained on these versions.
12
14
  > For more information see: https://techoverflow.net/2024/07/23/how-to-fix-numpy-dtype-size-changed-may-indicate-binary-incompatibility-expected-96-from-c-header-got-88-from-pyobject/
13
15
 
16
+ > [!NOTE]
17
+ > ValX will automatically install a version of `scikit-learn` that is compatible with your device if you don't have one already.
18
+
19
+ ## Changes in 0.2.5
20
+
21
+ ValX v0.2.5 introduces enhanced flexibility for profanity filtering by adding support for custom profanity lists:
22
+
23
+ - **Custom Profanity Word Lists**: Users can now provide their own lists of profane words directly as Python lists to the `detect_profanity` and `remove_profanity` functions via the new `custom_words_list` parameter.
24
+ - **Standalone Custom Lists**: Utilize your custom profanity list exclusively by setting the `language` parameter to `None`. ValX will then only use the words provided in `custom_words_list`.
25
+ - **Combined Lists**: Use a custom list in conjunction with ValX's built-in language-specific wordlists. Simply provide both a `language` (e.g., "English") and your `custom_words_list`. ValX will use the combined set of words.
26
+ - **Loading Custom Lists from File**: A new helper function, `load_custom_profanity_from_file(filepath)`, allows you to easily load custom profanity words from a text file.
27
+ - **File Format**: The file should contain one profanity word per line.
28
+ - Lines starting with a hash symbol (`#`) are treated as comments and ignored.
29
+ - Empty lines or lines containing only whitespace are also ignored.
30
+ - **Updated Detection Reporting**: The `detect_profanity` function's output now specifies the source of detected profanity more clearly (e.g., "Custom", "Custom + English").
31
+
32
+ These features give users greater control over the profanity filtering process, allowing for more tailored and specific use cases.
33
+
14
34
  ## Changes in 0.2.4
15
35
 
16
36
  Fixed a major incompatibility issue with `scikit-learn` due to version changes in `scikit-learn v1.3.0` which causes compatibility issues with versions later than `1.2.2`. ValX can now be used with `scikit-learn` versions earlier and later than `1.3.0`!
@@ -108,25 +128,129 @@ Below is a complete list of all the available supported languages for ValX's pro
108
128
 
109
129
  ## Usage
110
130
 
111
- ### Detect Profanity
131
+ ### Profanity Detection and Removal
132
+
133
+ ValX allows for flexible profanity filtering using built-in language lists, custom word lists (provided as Python lists or loaded from files), or a combination of both.
134
+
135
+ **1. Basic Profanity Detection (Built-in Language)**
112
136
 
113
137
  ```python
114
138
  from valx import detect_profanity
115
139
 
116
- # Detect profanity
140
+ sample_text = ["This is some fuck and porn text."]
141
+ # Detect profanity using the English list
117
142
  results = detect_profanity(sample_text, language='English')
118
- print("Profanity Evaluation Results", results)
143
+ # results will be:
144
+ # [
145
+ # {'Line': 1, 'Column': 14, 'Word': 'fuck', 'Language': 'English'},
146
+ # {'Line': 1, 'Column': 23, 'Word': 'porn', 'Language': 'English'}
147
+ # ]
148
+ print(results)
149
+ ```
150
+
151
+ **2. Profanity Detection with a Custom Word List (Python List)**
152
+
153
+ You can provide your own list of words to filter.
154
+
155
+ ```python
156
+ from valx import detect_profanity
157
+
158
+ sample_text = ["This contains custombadword1 and also asshole from English list."]
159
+ my_custom_words = ["custombadword1", "anothercustom"]
160
+
161
+ # Option A: Custom list ONLY (language=None)
162
+ results_custom_only = detect_profanity(sample_text, language=None, custom_words_list=my_custom_words)
163
+ # results_custom_only will detect "custombadword1" with Language: "Custom"
164
+ # [{'Line': 1, 'Column': 15, 'Word': 'custombadword1', 'Language': 'Custom'}]
165
+ print(results_custom_only)
166
+
167
+ # Option B: Custom list COMBINED with a built-in language
168
+ results_custom_plus_english = detect_profanity(sample_text, language="English", custom_words_list=my_custom_words)
169
+ # results_custom_plus_english will detect "custombadword1" and "asshole"
170
+ # Language will be "Custom + English"
171
+ # [
172
+ # {'Line': 1, 'Column': 15, 'Word': 'custombadword1', 'Language': 'Custom + English'},
173
+ # {'Line': 1, 'Column': 43, 'Word': 'asshole', 'Language': 'Custom + English'}
174
+ # ]
175
+ print(results_custom_plus_english)
119
176
  ```
120
177
 
121
- ### Remove Profanity
178
+ **3. Loading Custom Profanity Words from a File**
179
+
180
+ ValX provides a helper function to load words from a text file (one word per line, '#' for comments).
122
181
 
123
182
  ```python
124
- from valx import remove_profanity
183
+ from valx import detect_profanity, load_custom_profanity_from_file
184
+
185
+ # Assume 'my_profanity_file.txt' contains:
186
+ # customfileword1
187
+ # # this is a comment
188
+ # customfileword2
189
+
190
+ custom_words_from_file = load_custom_profanity_from_file("my_profanity_file.txt")
191
+ # custom_words_from_file will be: ['customfileword1', 'customfileword2']
192
+
193
+ sample_text_for_file = ["Text with customfileword1 and built-in shit."]
194
+
195
+ # Use file-loaded list with English built-in list
196
+ results_file_plus_english = detect_profanity(
197
+ sample_text_for_file,
198
+ language="English",
199
+ custom_words_list=custom_words_from_file
200
+ )
201
+ # Detects "customfileword1" and "shit", Language: "Custom + English"
202
+ print(results_file_plus_english)
203
+
204
+ # Use file-loaded list ONLY
205
+ results_file_only = detect_profanity(
206
+ sample_text_for_file,
207
+ language=None, # Important: set language to None
208
+ custom_words_list=custom_words_from_file
209
+ )
210
+ # Detects only "customfileword1", Language: "Custom"
211
+ print(results_file_only)
212
+ ```
213
+
214
+ **Output Format for `detect_profanity`**
125
215
 
126
- # Remove profanity
127
- removed = remove_profanity(sample_text, "text_cleaned.txt", language="English")
216
+ The `detect_profanity` function returns a list of dictionaries. Each dictionary includes:
217
+ - `"Line"`: The line number (1-indexed).
218
+ - `"Column"`: The column number (1-indexed) where the profanity starts.
219
+ - `"Word"`: The detected profanity word.
220
+ - `"Language"`: Indicates the source of the word list:
221
+ - `<LanguageName>` (e.g., "English"): If only a built-in language list was used.
222
+ - `"Custom"`: If `language=None` and only a `custom_words_list` was used.
223
+ - `"Custom + <LanguageName>"` (e.g., "Custom + English"): If both a built-in list and `custom_words_list` were used.
224
+ - `"Custom + All"`: If `language='All'` and `custom_words_list` were used.
225
+
226
+
227
+ **4. Removing Profanity**
228
+
229
+ `remove_profanity` works similarly, accepting `language` and `custom_words_list` parameters.
230
+
231
+ ```python
232
+ from valx import remove_profanity, load_custom_profanity_from_file
233
+
234
+ sample_text = ["This is fuck, custombadword1, and text with customfileword1."]
235
+ my_custom_words = ["custombadword1"]
236
+ custom_words_from_file = load_custom_profanity_from_file("my_profanity_file.txt") # Assuming it contains 'customfileword1'
237
+
238
+ # Remove profanity using English built-in + my_custom_words + custom_words_from_file
239
+ all_custom_words = list(set(my_custom_words + custom_words_from_file)) # Combine and unique
240
+
241
+ cleaned_text = remove_profanity(
242
+ sample_text,
243
+ output_file="cleaned_output.txt", # Optional: saves to file
244
+ language="English",
245
+ custom_words_list=all_custom_words
246
+ )
247
+ # cleaned_text will have "fuck", "custombadword1", and "customfileword1" replaced with "bad word".
248
+ # e.g., ["This is bad word, bad word, and text with bad word."]
249
+ print(cleaned_text)
128
250
  ```
129
251
 
252
+ The `load_profanity_words` function (used internally) also accepts `language` and `custom_words_list` if you need direct access to the word lists.
253
+
130
254
  ### Detect Sensitive Information
131
255
 
132
256
  ```python
@@ -5,7 +5,7 @@ with open("README.md", "r", encoding="utf-8") as fh:
5
5
 
6
6
  setup(
7
7
  name='valx',
8
- version='0.2.4',
8
+ version='0.2.5',
9
9
  author='Infinitode Pty Ltd',
10
10
  author_email='infinitode.ltd@gmail.com',
11
11
  description='An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.',
@@ -16,15 +16,20 @@ Creative Commons Attribution 4.0 International License:
16
16
  https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words/blob/master/LICENSE
17
17
  '''
18
18
 
19
- def load_profanity_words(language='English'):
19
+ def load_profanity_words(language='English', custom_words_list: list[str] = None):
20
20
  """
21
- Load profanity words for the specified language.
21
+ Load profanity words for the specified language, optionally including a custom list.
22
22
 
23
23
  Args:
24
- language (str): The language for which to load profanity words. Defaults to 'English'.
24
+ language (str, optional): The language for which to load profanity words.
25
+ Defaults to 'English'. Can be None to use only custom_words_list.
26
+ custom_words_list (list[str], optional): A list of custom profanity words. Defaults to None.
25
27
 
26
28
  Returns:
27
- list: A list of profanity words for the specified language, or all languages if 'All' is specified.
29
+ list: A list of profanity words.
30
+
31
+ Raises:
32
+ ValueError: If language is None and custom_words_list is None or empty.
28
33
  """
29
34
  # Profanity words as a string
30
35
  PROFANITY_WORDS = """
@@ -2740,35 +2745,102 @@ def load_profanity_words(language='English'):
2740
2745
  if current_language:
2741
2746
  profanity_lists[current_language] = current_words
2742
2747
 
2743
- if language == 'All':
2744
- all_profanity_words = []
2748
+ loaded_words = []
2749
+ if language is None:
2750
+ if not custom_words_list:
2751
+ raise ValueError("If language is None, custom_words_list must be provided and non-empty.")
2752
+ loaded_words = list(custom_words_list) # Use a copy
2753
+ elif language == 'All':
2745
2754
  for words in profanity_lists.values():
2746
- all_profanity_words.extend(words)
2747
- return all_profanity_words
2755
+ loaded_words.extend(words)
2756
+ if custom_words_list:
2757
+ loaded_words.extend(w for w in custom_words_list if w not in loaded_words)
2748
2758
  else:
2749
- return profanity_lists.get(language, [])
2759
+ loaded_words = list(profanity_lists.get(language, [])) # Use a copy
2760
+ if custom_words_list:
2761
+ loaded_words.extend(w for w in custom_words_list if w not in loaded_words)
2762
+
2763
+ return list(set(loaded_words)) # Return unique words
2750
2764
 
2751
2765
 
2752
- def detect_profanity(text_data, language='English'):
2766
+ def load_custom_profanity_from_file(filepath: str) -> list[str]:
2767
+ """
2768
+ Loads a custom list of profanity words from a file.
2769
+
2770
+ The file should contain one word per line. Lines starting with '#' will be ignored as comments.
2771
+ Empty lines or lines with only whitespace will also be ignored.
2772
+
2773
+ Args:
2774
+ filepath (str): The path to the file containing custom profanity words.
2775
+
2776
+ Returns:
2777
+ list[str]: A list of words loaded from the file.
2778
+
2779
+ Raises:
2780
+ FileNotFoundError: If the specified filepath does not exist.
2781
+ IOError: For other file reading issues.
2782
+ """
2783
+ custom_words = []
2784
+ try:
2785
+ with open(filepath, 'r', encoding='utf-8') as f:
2786
+ for line in f:
2787
+ word = line.strip()
2788
+ if word and not word.startswith('#'):
2789
+ custom_words.append(word)
2790
+ except FileNotFoundError:
2791
+ # Re-raise FileNotFoundError to be explicit, or handle as per specific project policy
2792
+ # For example, could log a warning and return empty list:
2793
+ # print(f"Warning: File not found at {filepath}. Returning empty custom word list.")
2794
+ # return []
2795
+ raise
2796
+ except IOError as e:
2797
+ # Handle other IO errors, perhaps log them or raise a more generic error
2798
+ # print(f"Warning: Error reading file {filepath}: {e}. Returning empty custom word list.")
2799
+ # return []
2800
+ raise IOError(f"Error reading custom profanity file {filepath}: {e}")
2801
+ return custom_words
2802
+
2803
+
2804
+ def detect_profanity(text_data, language='English', custom_words_list: list[str] = None):
2753
2805
  """
2754
2806
  Detect profanity in text data using regex.
2755
2807
 
2756
2808
  Args:
2757
2809
  text_data (list): A list of strings representing the text data to analyze.
2758
- language (str): The language used to detect profanity. Defaults to 'English'. Available languages include: All, Arabic, Czech, Danish, German, English, Esperanto, Persian, Finnish, Filipino, French, French (CA), Hindi, Hungarian, Italian, Japanese, Kabyle, Korean, Dutch, Norwegian, Polish, Portuguese, Russian, Swedish, Thai, Klingon, Turkish, Chinese.
2810
+ language (str, optional): The language used to detect profanity. Defaults to 'English'.
2811
+ Can be None if custom_words_list is provided.
2812
+ Available languages include: All, Arabic, Czech, Danish, German,
2813
+ English, Esperanto, Persian, Finnish, Filipino, French,
2814
+ French (CA), Hindi, Hungarian, Italian, Japanese, Kabyle,
2815
+ Korean, Dutch, Norwegian, Polish, Portuguese, Russian,
2816
+ Swedish, Thai, Klingon, Turkish, Chinese.
2817
+ custom_words_list (list[str], optional): A list of custom profanity words. Defaults to None.
2759
2818
  Returns:
2760
2819
  list: A list of dictionaries where each dictionary represents a detected instance of profanity.
2761
2820
  Each dictionary contains the following keys:
2762
2821
  - "Line" (int): The line number where the profanity was detected.
2763
2822
  - "Column" (int): The column number (position in the line) where the profanity starts.
2764
2823
  - "Word" (str): The detected profanity word.
2765
- - "Language" (str): The language in which the profanity was detected.
2824
+ - "Language" (str): The language string indicating the source of the profanity list.
2766
2825
  """
2767
- profanity_keywords = load_profanity_words(language)
2826
+ profanity_keywords = load_profanity_words(language, custom_words_list)
2768
2827
  detected = set() # Use a set to store unique occurrences
2769
2828
 
2829
+ display_language = "Custom"
2830
+ if language:
2831
+ if custom_words_list:
2832
+ display_language = f"Custom + {language}"
2833
+ else:
2834
+ display_language = language
2835
+ elif not custom_words_list:
2836
+ # This case should ideally be prevented by load_profanity_words raising an error,
2837
+ # but as a fallback, if somehow load_profanity_words returns empty and language is None
2838
+ # and no custom_words_list, we avoid error here.
2839
+ return []
2840
+
2841
+
2770
2842
  for i, line in enumerate(text_data):
2771
- # Skip lines with language markers
2843
+ # Skip lines with language markers (though less relevant if custom lists are primary)
2772
2844
  if line.startswith("$") and line.endswith("$"):
2773
2845
  continue
2774
2846
 
@@ -2776,36 +2848,39 @@ def detect_profanity(text_data, language='English'):
2776
2848
  for profanity in profanity_keywords:
2777
2849
  matches = re.finditer(r'\b{}\b'.format(re.escape(profanity)), line, flags=re.IGNORECASE)
2778
2850
  for match in matches:
2779
- detected.add((i + 1, match.start() + 1, language, profanity)) # Add to set
2851
+ # Store with display_language, not the input `language` parameter
2852
+ detected.add((i + 1, match.start() + 1, display_language, profanity))
2780
2853
 
2781
2854
  detections = [] # Initialize detections as a list
2782
2855
 
2783
2856
  if detected:
2784
- for line_num, col_num, lang, word in detected:
2857
+ for line_num, col_num, lang_display_name, word in detected:
2785
2858
  detection_info = {
2786
2859
  "Line": line_num,
2787
2860
  "Column": col_num,
2788
2861
  "Word": word,
2789
- "Language": lang
2862
+ "Language": lang_display_name
2790
2863
  }
2791
- detections.append(detection_info) # Append each detection to the list
2864
+ detections.append(detection_info)
2792
2865
 
2793
2866
  return detections
2794
2867
 
2795
2868
 
2796
- def remove_profanity(text_data, output_file=None, language='English'):
2869
+ def remove_profanity(text_data, output_file=None, language='English', custom_words_list: list[str] = None):
2797
2870
  """
2798
2871
  Remove profanity from text data.
2799
2872
 
2800
2873
  Args:
2801
2874
  text_data (list): A list of strings representing the text data to clean.
2802
- output_file (str): The file path to write the cleaned data. If None, cleaned data is not written to a file.
2803
- language (str): The language for which to remove profanity. Defaults to 'English'.
2875
+ output_file (str, optional): The file path to write the cleaned data. If None, cleaned data is not written to a file.
2876
+ language (str, optional): The language for which to remove profanity. Defaults to 'English'.
2877
+ Can be None if custom_words_list is provided.
2878
+ custom_words_list (list[str], optional): A list of custom profanity words. Defaults to None.
2804
2879
 
2805
2880
  Returns:
2806
2881
  list: A list of strings representing the cleaned text data.
2807
2882
  """
2808
- profanity_keywords = load_profanity_words(language)
2883
+ profanity_keywords = load_profanity_words(language, custom_words_list)
2809
2884
  cleaned_data = []
2810
2885
 
2811
2886
  for line in text_data:
@@ -1,6 +1,6 @@
1
- Metadata-Version: 2.1
1
+ Metadata-Version: 2.4
2
2
  Name: valx
3
- Version: 0.2.4
3
+ Version: 0.2.5
4
4
  Summary: An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.
5
5
  Home-page: https://github.com/infinitode/valx
6
6
  Author: Infinitode Pty Ltd
@@ -19,6 +19,16 @@ Requires-Python: >=3.6
19
19
  Description-Content-Type: text/markdown
20
20
  License-File: LICENSE
21
21
  Requires-Dist: scikit-learn
22
+ Dynamic: author
23
+ Dynamic: author-email
24
+ Dynamic: classifier
25
+ Dynamic: description
26
+ Dynamic: description-content-type
27
+ Dynamic: home-page
28
+ Dynamic: license-file
29
+ Dynamic: requires-dist
30
+ Dynamic: requires-python
31
+ Dynamic: summary
22
32
 
23
33
  # ValX
24
34
  ![Python Version](https://img.shields.io/badge/python-3.12-blue.svg)
@@ -30,9 +40,29 @@ Requires-Dist: scikit-learn
30
40
  An open-source Python library for data cleaning tasks. It includes functions for profanity detection, and removal, and detection and removal of personal information. Also includes hate speech and offensive language detection and removal, using AI.
31
41
 
32
42
  > [!IMPORTANT]
43
+ > If you are using `scikit-learn` versions older than version `1.3.0`, please also downgrade your version of `numpy` as stated below. Otherwise, you can continue to use your preferred version of `scikit-learn` without downgrading `numpy`.
44
+ >
33
45
  > Please downgrade to `numpy` version `1.26.4`. Our ValX **DecisionTreeClassifier** AI model, relies on lower versions of `numpy`, because it was trained on these versions.
34
46
  > For more information see: https://techoverflow.net/2024/07/23/how-to-fix-numpy-dtype-size-changed-may-indicate-binary-incompatibility-expected-96-from-c-header-got-88-from-pyobject/
35
47
 
48
+ > [!NOTE]
49
+ > ValX will automatically install a version of `scikit-learn` that is compatible with your device if you don't have one already.
50
+
51
+ ## Changes in 0.2.5
52
+
53
+ ValX v0.2.5 introduces enhanced flexibility for profanity filtering by adding support for custom profanity lists:
54
+
55
+ - **Custom Profanity Word Lists**: Users can now provide their own lists of profane words directly as Python lists to the `detect_profanity` and `remove_profanity` functions via the new `custom_words_list` parameter.
56
+ - **Standalone Custom Lists**: Utilize your custom profanity list exclusively by setting the `language` parameter to `None`. ValX will then only use the words provided in `custom_words_list`.
57
+ - **Combined Lists**: Use a custom list in conjunction with ValX's built-in language-specific wordlists. Simply provide both a `language` (e.g., "English") and your `custom_words_list`. ValX will use the combined set of words.
58
+ - **Loading Custom Lists from File**: A new helper function, `load_custom_profanity_from_file(filepath)`, allows you to easily load custom profanity words from a text file.
59
+ - **File Format**: The file should contain one profanity word per line.
60
+ - Lines starting with a hash symbol (`#`) are treated as comments and ignored.
61
+ - Empty lines or lines containing only whitespace are also ignored.
62
+ - **Updated Detection Reporting**: The `detect_profanity` function's output now specifies the source of detected profanity more clearly (e.g., "Custom", "Custom + English").
63
+
64
+ These features give users greater control over the profanity filtering process, allowing for more tailored and specific use cases.
65
+
36
66
  ## Changes in 0.2.4
37
67
 
38
68
  Fixed a major incompatibility issue with `scikit-learn` due to version changes in `scikit-learn v1.3.0` which causes compatibility issues with versions later than `1.2.2`. ValX can now be used with `scikit-learn` versions earlier and later than `1.3.0`!
@@ -130,25 +160,129 @@ Below is a complete list of all the available supported languages for ValX's pro
130
160
 
131
161
  ## Usage
132
162
 
133
- ### Detect Profanity
163
+ ### Profanity Detection and Removal
164
+
165
+ ValX allows for flexible profanity filtering using built-in language lists, custom word lists (provided as Python lists or loaded from files), or a combination of both.
166
+
167
+ **1. Basic Profanity Detection (Built-in Language)**
134
168
 
135
169
  ```python
136
170
  from valx import detect_profanity
137
171
 
138
- # Detect profanity
172
+ sample_text = ["This is some fuck and porn text."]
173
+ # Detect profanity using the English list
139
174
  results = detect_profanity(sample_text, language='English')
140
- print("Profanity Evaluation Results", results)
175
+ # results will be:
176
+ # [
177
+ # {'Line': 1, 'Column': 14, 'Word': 'fuck', 'Language': 'English'},
178
+ # {'Line': 1, 'Column': 23, 'Word': 'porn', 'Language': 'English'}
179
+ # ]
180
+ print(results)
181
+ ```
182
+
183
+ **2. Profanity Detection with a Custom Word List (Python List)**
184
+
185
+ You can provide your own list of words to filter.
186
+
187
+ ```python
188
+ from valx import detect_profanity
189
+
190
+ sample_text = ["This contains custombadword1 and also asshole from English list."]
191
+ my_custom_words = ["custombadword1", "anothercustom"]
192
+
193
+ # Option A: Custom list ONLY (language=None)
194
+ results_custom_only = detect_profanity(sample_text, language=None, custom_words_list=my_custom_words)
195
+ # results_custom_only will detect "custombadword1" with Language: "Custom"
196
+ # [{'Line': 1, 'Column': 15, 'Word': 'custombadword1', 'Language': 'Custom'}]
197
+ print(results_custom_only)
198
+
199
+ # Option B: Custom list COMBINED with a built-in language
200
+ results_custom_plus_english = detect_profanity(sample_text, language="English", custom_words_list=my_custom_words)
201
+ # results_custom_plus_english will detect "custombadword1" and "asshole"
202
+ # Language will be "Custom + English"
203
+ # [
204
+ # {'Line': 1, 'Column': 15, 'Word': 'custombadword1', 'Language': 'Custom + English'},
205
+ # {'Line': 1, 'Column': 43, 'Word': 'asshole', 'Language': 'Custom + English'}
206
+ # ]
207
+ print(results_custom_plus_english)
141
208
  ```
142
209
 
143
- ### Remove Profanity
210
+ **3. Loading Custom Profanity Words from a File**
211
+
212
+ ValX provides a helper function to load words from a text file (one word per line, '#' for comments).
144
213
 
145
214
  ```python
146
- from valx import remove_profanity
215
+ from valx import detect_profanity, load_custom_profanity_from_file
216
+
217
+ # Assume 'my_profanity_file.txt' contains:
218
+ # customfileword1
219
+ # # this is a comment
220
+ # customfileword2
221
+
222
+ custom_words_from_file = load_custom_profanity_from_file("my_profanity_file.txt")
223
+ # custom_words_from_file will be: ['customfileword1', 'customfileword2']
224
+
225
+ sample_text_for_file = ["Text with customfileword1 and built-in shit."]
226
+
227
+ # Use file-loaded list with English built-in list
228
+ results_file_plus_english = detect_profanity(
229
+ sample_text_for_file,
230
+ language="English",
231
+ custom_words_list=custom_words_from_file
232
+ )
233
+ # Detects "customfileword1" and "shit", Language: "Custom + English"
234
+ print(results_file_plus_english)
235
+
236
+ # Use file-loaded list ONLY
237
+ results_file_only = detect_profanity(
238
+ sample_text_for_file,
239
+ language=None, # Important: set language to None
240
+ custom_words_list=custom_words_from_file
241
+ )
242
+ # Detects only "customfileword1", Language: "Custom"
243
+ print(results_file_only)
244
+ ```
245
+
246
+ **Output Format for `detect_profanity`**
147
247
 
148
- # Remove profanity
149
- removed = remove_profanity(sample_text, "text_cleaned.txt", language="English")
248
+ The `detect_profanity` function returns a list of dictionaries. Each dictionary includes:
249
+ - `"Line"`: The line number (1-indexed).
250
+ - `"Column"`: The column number (1-indexed) where the profanity starts.
251
+ - `"Word"`: The detected profanity word.
252
+ - `"Language"`: Indicates the source of the word list:
253
+ - `<LanguageName>` (e.g., "English"): If only a built-in language list was used.
254
+ - `"Custom"`: If `language=None` and only a `custom_words_list` was used.
255
+ - `"Custom + <LanguageName>"` (e.g., "Custom + English"): If both a built-in list and `custom_words_list` were used.
256
+ - `"Custom + All"`: If `language='All'` and `custom_words_list` were used.
257
+
258
+
259
+ **4. Removing Profanity**
260
+
261
+ `remove_profanity` works similarly, accepting `language` and `custom_words_list` parameters.
262
+
263
+ ```python
264
+ from valx import remove_profanity, load_custom_profanity_from_file
265
+
266
+ sample_text = ["This is fuck, custombadword1, and text with customfileword1."]
267
+ my_custom_words = ["custombadword1"]
268
+ custom_words_from_file = load_custom_profanity_from_file("my_profanity_file.txt") # Assuming it contains 'customfileword1'
269
+
270
+ # Remove profanity using English built-in + my_custom_words + custom_words_from_file
271
+ all_custom_words = list(set(my_custom_words + custom_words_from_file)) # Combine and unique
272
+
273
+ cleaned_text = remove_profanity(
274
+ sample_text,
275
+ output_file="cleaned_output.txt", # Optional: saves to file
276
+ language="English",
277
+ custom_words_list=all_custom_words
278
+ )
279
+ # cleaned_text will have "fuck", "custombadword1", and "customfileword1" replaced with "bad word".
280
+ # e.g., ["This is bad word, bad word, and text with bad word."]
281
+ print(cleaned_text)
150
282
  ```
151
283
 
284
+ The `load_profanity_words` function (used internally) also accepts `language` and `custom_words_list` if you need direct access to the word lists.
285
+
152
286
  ### Detect Sensitive Information
153
287
 
154
288
  ```python
File without changes
File without changes
File without changes
File without changes
File without changes