valx 0.2.4__tar.gz → 0.2.5__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {valx-0.2.4/valx.egg-info → valx-0.2.5}/PKG-INFO +143 -9
- {valx-0.2.4 → valx-0.2.5}/README.md +131 -7
- {valx-0.2.4 → valx-0.2.5}/setup.py +1 -1
- {valx-0.2.4 → valx-0.2.5}/valx/__init__.py +97 -22
- {valx-0.2.4 → valx-0.2.5/valx.egg-info}/PKG-INFO +143 -9
- {valx-0.2.4 → valx-0.2.5}/LICENSE +0 -0
- {valx-0.2.4 → valx-0.2.5}/setup.cfg +0 -0
- {valx-0.2.4 → valx-0.2.5}/valx/models/classifier_upgraded.pkl +0 -0
- {valx-0.2.4 → valx-0.2.5}/valx/models/count_vectorizer.sav +0 -0
- {valx-0.2.4 → valx-0.2.5}/valx/models/decision_tree_model.sav +0 -0
- {valx-0.2.4 → valx-0.2.5}/valx.egg-info/SOURCES.txt +0 -0
- {valx-0.2.4 → valx-0.2.5}/valx.egg-info/dependency_links.txt +0 -0
- {valx-0.2.4 → valx-0.2.5}/valx.egg-info/requires.txt +0 -0
- {valx-0.2.4 → valx-0.2.5}/valx.egg-info/top_level.txt +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
|
-
Metadata-Version: 2.
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
2
|
Name: valx
|
|
3
|
-
Version: 0.2.
|
|
3
|
+
Version: 0.2.5
|
|
4
4
|
Summary: An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.
|
|
5
5
|
Home-page: https://github.com/infinitode/valx
|
|
6
6
|
Author: Infinitode Pty Ltd
|
|
@@ -19,6 +19,16 @@ Requires-Python: >=3.6
|
|
|
19
19
|
Description-Content-Type: text/markdown
|
|
20
20
|
License-File: LICENSE
|
|
21
21
|
Requires-Dist: scikit-learn
|
|
22
|
+
Dynamic: author
|
|
23
|
+
Dynamic: author-email
|
|
24
|
+
Dynamic: classifier
|
|
25
|
+
Dynamic: description
|
|
26
|
+
Dynamic: description-content-type
|
|
27
|
+
Dynamic: home-page
|
|
28
|
+
Dynamic: license-file
|
|
29
|
+
Dynamic: requires-dist
|
|
30
|
+
Dynamic: requires-python
|
|
31
|
+
Dynamic: summary
|
|
22
32
|
|
|
23
33
|
# ValX
|
|
24
34
|

|
|
@@ -30,9 +40,29 @@ Requires-Dist: scikit-learn
|
|
|
30
40
|
An open-source Python library for data cleaning tasks. It includes functions for profanity detection, and removal, and detection and removal of personal information. Also includes hate speech and offensive language detection and removal, using AI.
|
|
31
41
|
|
|
32
42
|
> [!IMPORTANT]
|
|
43
|
+
> If you are using `scikit-learn` versions older than version `1.3.0`, please also downgrade your version of `numpy` as stated below. Otherwise, you can continue to use your preferred version of `scikit-learn` without downgrading `numpy`.
|
|
44
|
+
>
|
|
33
45
|
> Please downgrade to `numpy` version `1.26.4`. Our ValX **DecisionTreeClassifier** AI model, relies on lower versions of `numpy`, because it was trained on these versions.
|
|
34
46
|
> For more information see: https://techoverflow.net/2024/07/23/how-to-fix-numpy-dtype-size-changed-may-indicate-binary-incompatibility-expected-96-from-c-header-got-88-from-pyobject/
|
|
35
47
|
|
|
48
|
+
> [!NOTE]
|
|
49
|
+
> ValX will automatically install a version of `scikit-learn` that is compatible with your device if you don't have one already.
|
|
50
|
+
|
|
51
|
+
## Changes in 0.2.5
|
|
52
|
+
|
|
53
|
+
ValX v0.2.5 introduces enhanced flexibility for profanity filtering by adding support for custom profanity lists:
|
|
54
|
+
|
|
55
|
+
- **Custom Profanity Word Lists**: Users can now provide their own lists of profane words directly as Python lists to the `detect_profanity` and `remove_profanity` functions via the new `custom_words_list` parameter.
|
|
56
|
+
- **Standalone Custom Lists**: Utilize your custom profanity list exclusively by setting the `language` parameter to `None`. ValX will then only use the words provided in `custom_words_list`.
|
|
57
|
+
- **Combined Lists**: Use a custom list in conjunction with ValX's built-in language-specific wordlists. Simply provide both a `language` (e.g., "English") and your `custom_words_list`. ValX will use the combined set of words.
|
|
58
|
+
- **Loading Custom Lists from File**: A new helper function, `load_custom_profanity_from_file(filepath)`, allows you to easily load custom profanity words from a text file.
|
|
59
|
+
- **File Format**: The file should contain one profanity word per line.
|
|
60
|
+
- Lines starting with a hash symbol (`#`) are treated as comments and ignored.
|
|
61
|
+
- Empty lines or lines containing only whitespace are also ignored.
|
|
62
|
+
- **Updated Detection Reporting**: The `detect_profanity` function's output now specifies the source of detected profanity more clearly (e.g., "Custom", "Custom + English").
|
|
63
|
+
|
|
64
|
+
These features give users greater control over the profanity filtering process, allowing for more tailored and specific use cases.
|
|
65
|
+
|
|
36
66
|
## Changes in 0.2.4
|
|
37
67
|
|
|
38
68
|
Fixed a major incompatibility issue with `scikit-learn` due to version changes in `scikit-learn v1.3.0` which causes compatibility issues with versions later than `1.2.2`. ValX can now be used with `scikit-learn` versions earlier and later than `1.3.0`!
|
|
@@ -130,25 +160,129 @@ Below is a complete list of all the available supported languages for ValX's pro
|
|
|
130
160
|
|
|
131
161
|
## Usage
|
|
132
162
|
|
|
133
|
-
###
|
|
163
|
+
### Profanity Detection and Removal
|
|
164
|
+
|
|
165
|
+
ValX allows for flexible profanity filtering using built-in language lists, custom word lists (provided as Python lists or loaded from files), or a combination of both.
|
|
166
|
+
|
|
167
|
+
**1. Basic Profanity Detection (Built-in Language)**
|
|
134
168
|
|
|
135
169
|
```python
|
|
136
170
|
from valx import detect_profanity
|
|
137
171
|
|
|
138
|
-
|
|
172
|
+
sample_text = ["This is some fuck and porn text."]
|
|
173
|
+
# Detect profanity using the English list
|
|
139
174
|
results = detect_profanity(sample_text, language='English')
|
|
140
|
-
|
|
175
|
+
# results will be:
|
|
176
|
+
# [
|
|
177
|
+
# {'Line': 1, 'Column': 14, 'Word': 'fuck', 'Language': 'English'},
|
|
178
|
+
# {'Line': 1, 'Column': 23, 'Word': 'porn', 'Language': 'English'}
|
|
179
|
+
# ]
|
|
180
|
+
print(results)
|
|
181
|
+
```
|
|
182
|
+
|
|
183
|
+
**2. Profanity Detection with a Custom Word List (Python List)**
|
|
184
|
+
|
|
185
|
+
You can provide your own list of words to filter.
|
|
186
|
+
|
|
187
|
+
```python
|
|
188
|
+
from valx import detect_profanity
|
|
189
|
+
|
|
190
|
+
sample_text = ["This contains custombadword1 and also asshole from English list."]
|
|
191
|
+
my_custom_words = ["custombadword1", "anothercustom"]
|
|
192
|
+
|
|
193
|
+
# Option A: Custom list ONLY (language=None)
|
|
194
|
+
results_custom_only = detect_profanity(sample_text, language=None, custom_words_list=my_custom_words)
|
|
195
|
+
# results_custom_only will detect "custombadword1" with Language: "Custom"
|
|
196
|
+
# [{'Line': 1, 'Column': 15, 'Word': 'custombadword1', 'Language': 'Custom'}]
|
|
197
|
+
print(results_custom_only)
|
|
198
|
+
|
|
199
|
+
# Option B: Custom list COMBINED with a built-in language
|
|
200
|
+
results_custom_plus_english = detect_profanity(sample_text, language="English", custom_words_list=my_custom_words)
|
|
201
|
+
# results_custom_plus_english will detect "custombadword1" and "asshole"
|
|
202
|
+
# Language will be "Custom + English"
|
|
203
|
+
# [
|
|
204
|
+
# {'Line': 1, 'Column': 15, 'Word': 'custombadword1', 'Language': 'Custom + English'},
|
|
205
|
+
# {'Line': 1, 'Column': 43, 'Word': 'asshole', 'Language': 'Custom + English'}
|
|
206
|
+
# ]
|
|
207
|
+
print(results_custom_plus_english)
|
|
141
208
|
```
|
|
142
209
|
|
|
143
|
-
|
|
210
|
+
**3. Loading Custom Profanity Words from a File**
|
|
211
|
+
|
|
212
|
+
ValX provides a helper function to load words from a text file (one word per line, '#' for comments).
|
|
144
213
|
|
|
145
214
|
```python
|
|
146
|
-
from valx import
|
|
215
|
+
from valx import detect_profanity, load_custom_profanity_from_file
|
|
216
|
+
|
|
217
|
+
# Assume 'my_profanity_file.txt' contains:
|
|
218
|
+
# customfileword1
|
|
219
|
+
# # this is a comment
|
|
220
|
+
# customfileword2
|
|
221
|
+
|
|
222
|
+
custom_words_from_file = load_custom_profanity_from_file("my_profanity_file.txt")
|
|
223
|
+
# custom_words_from_file will be: ['customfileword1', 'customfileword2']
|
|
224
|
+
|
|
225
|
+
sample_text_for_file = ["Text with customfileword1 and built-in shit."]
|
|
226
|
+
|
|
227
|
+
# Use file-loaded list with English built-in list
|
|
228
|
+
results_file_plus_english = detect_profanity(
|
|
229
|
+
sample_text_for_file,
|
|
230
|
+
language="English",
|
|
231
|
+
custom_words_list=custom_words_from_file
|
|
232
|
+
)
|
|
233
|
+
# Detects "customfileword1" and "shit", Language: "Custom + English"
|
|
234
|
+
print(results_file_plus_english)
|
|
235
|
+
|
|
236
|
+
# Use file-loaded list ONLY
|
|
237
|
+
results_file_only = detect_profanity(
|
|
238
|
+
sample_text_for_file,
|
|
239
|
+
language=None, # Important: set language to None
|
|
240
|
+
custom_words_list=custom_words_from_file
|
|
241
|
+
)
|
|
242
|
+
# Detects only "customfileword1", Language: "Custom"
|
|
243
|
+
print(results_file_only)
|
|
244
|
+
```
|
|
245
|
+
|
|
246
|
+
**Output Format for `detect_profanity`**
|
|
147
247
|
|
|
148
|
-
|
|
149
|
-
|
|
248
|
+
The `detect_profanity` function returns a list of dictionaries. Each dictionary includes:
|
|
249
|
+
- `"Line"`: The line number (1-indexed).
|
|
250
|
+
- `"Column"`: The column number (1-indexed) where the profanity starts.
|
|
251
|
+
- `"Word"`: The detected profanity word.
|
|
252
|
+
- `"Language"`: Indicates the source of the word list:
|
|
253
|
+
- `<LanguageName>` (e.g., "English"): If only a built-in language list was used.
|
|
254
|
+
- `"Custom"`: If `language=None` and only a `custom_words_list` was used.
|
|
255
|
+
- `"Custom + <LanguageName>"` (e.g., "Custom + English"): If both a built-in list and `custom_words_list` were used.
|
|
256
|
+
- `"Custom + All"`: If `language='All'` and `custom_words_list` were used.
|
|
257
|
+
|
|
258
|
+
|
|
259
|
+
**4. Removing Profanity**
|
|
260
|
+
|
|
261
|
+
`remove_profanity` works similarly, accepting `language` and `custom_words_list` parameters.
|
|
262
|
+
|
|
263
|
+
```python
|
|
264
|
+
from valx import remove_profanity, load_custom_profanity_from_file
|
|
265
|
+
|
|
266
|
+
sample_text = ["This is fuck, custombadword1, and text with customfileword1."]
|
|
267
|
+
my_custom_words = ["custombadword1"]
|
|
268
|
+
custom_words_from_file = load_custom_profanity_from_file("my_profanity_file.txt") # Assuming it contains 'customfileword1'
|
|
269
|
+
|
|
270
|
+
# Remove profanity using English built-in + my_custom_words + custom_words_from_file
|
|
271
|
+
all_custom_words = list(set(my_custom_words + custom_words_from_file)) # Combine and unique
|
|
272
|
+
|
|
273
|
+
cleaned_text = remove_profanity(
|
|
274
|
+
sample_text,
|
|
275
|
+
output_file="cleaned_output.txt", # Optional: saves to file
|
|
276
|
+
language="English",
|
|
277
|
+
custom_words_list=all_custom_words
|
|
278
|
+
)
|
|
279
|
+
# cleaned_text will have "fuck", "custombadword1", and "customfileword1" replaced with "bad word".
|
|
280
|
+
# e.g., ["This is bad word, bad word, and text with bad word."]
|
|
281
|
+
print(cleaned_text)
|
|
150
282
|
```
|
|
151
283
|
|
|
284
|
+
The `load_profanity_words` function (used internally) also accepts `language` and `custom_words_list` if you need direct access to the word lists.
|
|
285
|
+
|
|
152
286
|
### Detect Sensitive Information
|
|
153
287
|
|
|
154
288
|
```python
|
|
@@ -8,9 +8,29 @@
|
|
|
8
8
|
An open-source Python library for data cleaning tasks. It includes functions for profanity detection, and removal, and detection and removal of personal information. Also includes hate speech and offensive language detection and removal, using AI.
|
|
9
9
|
|
|
10
10
|
> [!IMPORTANT]
|
|
11
|
+
> If you are using `scikit-learn` versions older than version `1.3.0`, please also downgrade your version of `numpy` as stated below. Otherwise, you can continue to use your preferred version of `scikit-learn` without downgrading `numpy`.
|
|
12
|
+
>
|
|
11
13
|
> Please downgrade to `numpy` version `1.26.4`. Our ValX **DecisionTreeClassifier** AI model, relies on lower versions of `numpy`, because it was trained on these versions.
|
|
12
14
|
> For more information see: https://techoverflow.net/2024/07/23/how-to-fix-numpy-dtype-size-changed-may-indicate-binary-incompatibility-expected-96-from-c-header-got-88-from-pyobject/
|
|
13
15
|
|
|
16
|
+
> [!NOTE]
|
|
17
|
+
> ValX will automatically install a version of `scikit-learn` that is compatible with your device if you don't have one already.
|
|
18
|
+
|
|
19
|
+
## Changes in 0.2.5
|
|
20
|
+
|
|
21
|
+
ValX v0.2.5 introduces enhanced flexibility for profanity filtering by adding support for custom profanity lists:
|
|
22
|
+
|
|
23
|
+
- **Custom Profanity Word Lists**: Users can now provide their own lists of profane words directly as Python lists to the `detect_profanity` and `remove_profanity` functions via the new `custom_words_list` parameter.
|
|
24
|
+
- **Standalone Custom Lists**: Utilize your custom profanity list exclusively by setting the `language` parameter to `None`. ValX will then only use the words provided in `custom_words_list`.
|
|
25
|
+
- **Combined Lists**: Use a custom list in conjunction with ValX's built-in language-specific wordlists. Simply provide both a `language` (e.g., "English") and your `custom_words_list`. ValX will use the combined set of words.
|
|
26
|
+
- **Loading Custom Lists from File**: A new helper function, `load_custom_profanity_from_file(filepath)`, allows you to easily load custom profanity words from a text file.
|
|
27
|
+
- **File Format**: The file should contain one profanity word per line.
|
|
28
|
+
- Lines starting with a hash symbol (`#`) are treated as comments and ignored.
|
|
29
|
+
- Empty lines or lines containing only whitespace are also ignored.
|
|
30
|
+
- **Updated Detection Reporting**: The `detect_profanity` function's output now specifies the source of detected profanity more clearly (e.g., "Custom", "Custom + English").
|
|
31
|
+
|
|
32
|
+
These features give users greater control over the profanity filtering process, allowing for more tailored and specific use cases.
|
|
33
|
+
|
|
14
34
|
## Changes in 0.2.4
|
|
15
35
|
|
|
16
36
|
Fixed a major incompatibility issue with `scikit-learn` due to version changes in `scikit-learn v1.3.0` which causes compatibility issues with versions later than `1.2.2`. ValX can now be used with `scikit-learn` versions earlier and later than `1.3.0`!
|
|
@@ -108,25 +128,129 @@ Below is a complete list of all the available supported languages for ValX's pro
|
|
|
108
128
|
|
|
109
129
|
## Usage
|
|
110
130
|
|
|
111
|
-
###
|
|
131
|
+
### Profanity Detection and Removal
|
|
132
|
+
|
|
133
|
+
ValX allows for flexible profanity filtering using built-in language lists, custom word lists (provided as Python lists or loaded from files), or a combination of both.
|
|
134
|
+
|
|
135
|
+
**1. Basic Profanity Detection (Built-in Language)**
|
|
112
136
|
|
|
113
137
|
```python
|
|
114
138
|
from valx import detect_profanity
|
|
115
139
|
|
|
116
|
-
|
|
140
|
+
sample_text = ["This is some fuck and porn text."]
|
|
141
|
+
# Detect profanity using the English list
|
|
117
142
|
results = detect_profanity(sample_text, language='English')
|
|
118
|
-
|
|
143
|
+
# results will be:
|
|
144
|
+
# [
|
|
145
|
+
# {'Line': 1, 'Column': 14, 'Word': 'fuck', 'Language': 'English'},
|
|
146
|
+
# {'Line': 1, 'Column': 23, 'Word': 'porn', 'Language': 'English'}
|
|
147
|
+
# ]
|
|
148
|
+
print(results)
|
|
149
|
+
```
|
|
150
|
+
|
|
151
|
+
**2. Profanity Detection with a Custom Word List (Python List)**
|
|
152
|
+
|
|
153
|
+
You can provide your own list of words to filter.
|
|
154
|
+
|
|
155
|
+
```python
|
|
156
|
+
from valx import detect_profanity
|
|
157
|
+
|
|
158
|
+
sample_text = ["This contains custombadword1 and also asshole from English list."]
|
|
159
|
+
my_custom_words = ["custombadword1", "anothercustom"]
|
|
160
|
+
|
|
161
|
+
# Option A: Custom list ONLY (language=None)
|
|
162
|
+
results_custom_only = detect_profanity(sample_text, language=None, custom_words_list=my_custom_words)
|
|
163
|
+
# results_custom_only will detect "custombadword1" with Language: "Custom"
|
|
164
|
+
# [{'Line': 1, 'Column': 15, 'Word': 'custombadword1', 'Language': 'Custom'}]
|
|
165
|
+
print(results_custom_only)
|
|
166
|
+
|
|
167
|
+
# Option B: Custom list COMBINED with a built-in language
|
|
168
|
+
results_custom_plus_english = detect_profanity(sample_text, language="English", custom_words_list=my_custom_words)
|
|
169
|
+
# results_custom_plus_english will detect "custombadword1" and "asshole"
|
|
170
|
+
# Language will be "Custom + English"
|
|
171
|
+
# [
|
|
172
|
+
# {'Line': 1, 'Column': 15, 'Word': 'custombadword1', 'Language': 'Custom + English'},
|
|
173
|
+
# {'Line': 1, 'Column': 43, 'Word': 'asshole', 'Language': 'Custom + English'}
|
|
174
|
+
# ]
|
|
175
|
+
print(results_custom_plus_english)
|
|
119
176
|
```
|
|
120
177
|
|
|
121
|
-
|
|
178
|
+
**3. Loading Custom Profanity Words from a File**
|
|
179
|
+
|
|
180
|
+
ValX provides a helper function to load words from a text file (one word per line, '#' for comments).
|
|
122
181
|
|
|
123
182
|
```python
|
|
124
|
-
from valx import
|
|
183
|
+
from valx import detect_profanity, load_custom_profanity_from_file
|
|
184
|
+
|
|
185
|
+
# Assume 'my_profanity_file.txt' contains:
|
|
186
|
+
# customfileword1
|
|
187
|
+
# # this is a comment
|
|
188
|
+
# customfileword2
|
|
189
|
+
|
|
190
|
+
custom_words_from_file = load_custom_profanity_from_file("my_profanity_file.txt")
|
|
191
|
+
# custom_words_from_file will be: ['customfileword1', 'customfileword2']
|
|
192
|
+
|
|
193
|
+
sample_text_for_file = ["Text with customfileword1 and built-in shit."]
|
|
194
|
+
|
|
195
|
+
# Use file-loaded list with English built-in list
|
|
196
|
+
results_file_plus_english = detect_profanity(
|
|
197
|
+
sample_text_for_file,
|
|
198
|
+
language="English",
|
|
199
|
+
custom_words_list=custom_words_from_file
|
|
200
|
+
)
|
|
201
|
+
# Detects "customfileword1" and "shit", Language: "Custom + English"
|
|
202
|
+
print(results_file_plus_english)
|
|
203
|
+
|
|
204
|
+
# Use file-loaded list ONLY
|
|
205
|
+
results_file_only = detect_profanity(
|
|
206
|
+
sample_text_for_file,
|
|
207
|
+
language=None, # Important: set language to None
|
|
208
|
+
custom_words_list=custom_words_from_file
|
|
209
|
+
)
|
|
210
|
+
# Detects only "customfileword1", Language: "Custom"
|
|
211
|
+
print(results_file_only)
|
|
212
|
+
```
|
|
213
|
+
|
|
214
|
+
**Output Format for `detect_profanity`**
|
|
125
215
|
|
|
126
|
-
|
|
127
|
-
|
|
216
|
+
The `detect_profanity` function returns a list of dictionaries. Each dictionary includes:
|
|
217
|
+
- `"Line"`: The line number (1-indexed).
|
|
218
|
+
- `"Column"`: The column number (1-indexed) where the profanity starts.
|
|
219
|
+
- `"Word"`: The detected profanity word.
|
|
220
|
+
- `"Language"`: Indicates the source of the word list:
|
|
221
|
+
- `<LanguageName>` (e.g., "English"): If only a built-in language list was used.
|
|
222
|
+
- `"Custom"`: If `language=None` and only a `custom_words_list` was used.
|
|
223
|
+
- `"Custom + <LanguageName>"` (e.g., "Custom + English"): If both a built-in list and `custom_words_list` were used.
|
|
224
|
+
- `"Custom + All"`: If `language='All'` and `custom_words_list` were used.
|
|
225
|
+
|
|
226
|
+
|
|
227
|
+
**4. Removing Profanity**
|
|
228
|
+
|
|
229
|
+
`remove_profanity` works similarly, accepting `language` and `custom_words_list` parameters.
|
|
230
|
+
|
|
231
|
+
```python
|
|
232
|
+
from valx import remove_profanity, load_custom_profanity_from_file
|
|
233
|
+
|
|
234
|
+
sample_text = ["This is fuck, custombadword1, and text with customfileword1."]
|
|
235
|
+
my_custom_words = ["custombadword1"]
|
|
236
|
+
custom_words_from_file = load_custom_profanity_from_file("my_profanity_file.txt") # Assuming it contains 'customfileword1'
|
|
237
|
+
|
|
238
|
+
# Remove profanity using English built-in + my_custom_words + custom_words_from_file
|
|
239
|
+
all_custom_words = list(set(my_custom_words + custom_words_from_file)) # Combine and unique
|
|
240
|
+
|
|
241
|
+
cleaned_text = remove_profanity(
|
|
242
|
+
sample_text,
|
|
243
|
+
output_file="cleaned_output.txt", # Optional: saves to file
|
|
244
|
+
language="English",
|
|
245
|
+
custom_words_list=all_custom_words
|
|
246
|
+
)
|
|
247
|
+
# cleaned_text will have "fuck", "custombadword1", and "customfileword1" replaced with "bad word".
|
|
248
|
+
# e.g., ["This is bad word, bad word, and text with bad word."]
|
|
249
|
+
print(cleaned_text)
|
|
128
250
|
```
|
|
129
251
|
|
|
252
|
+
The `load_profanity_words` function (used internally) also accepts `language` and `custom_words_list` if you need direct access to the word lists.
|
|
253
|
+
|
|
130
254
|
### Detect Sensitive Information
|
|
131
255
|
|
|
132
256
|
```python
|
|
@@ -5,7 +5,7 @@ with open("README.md", "r", encoding="utf-8") as fh:
|
|
|
5
5
|
|
|
6
6
|
setup(
|
|
7
7
|
name='valx',
|
|
8
|
-
version='0.2.
|
|
8
|
+
version='0.2.5',
|
|
9
9
|
author='Infinitode Pty Ltd',
|
|
10
10
|
author_email='infinitode.ltd@gmail.com',
|
|
11
11
|
description='An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.',
|
|
@@ -16,15 +16,20 @@ Creative Commons Attribution 4.0 International License:
|
|
|
16
16
|
https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words/blob/master/LICENSE
|
|
17
17
|
'''
|
|
18
18
|
|
|
19
|
-
def load_profanity_words(language='English'):
|
|
19
|
+
def load_profanity_words(language='English', custom_words_list: list[str] = None):
|
|
20
20
|
"""
|
|
21
|
-
Load profanity words for the specified language.
|
|
21
|
+
Load profanity words for the specified language, optionally including a custom list.
|
|
22
22
|
|
|
23
23
|
Args:
|
|
24
|
-
language (str): The language for which to load profanity words.
|
|
24
|
+
language (str, optional): The language for which to load profanity words.
|
|
25
|
+
Defaults to 'English'. Can be None to use only custom_words_list.
|
|
26
|
+
custom_words_list (list[str], optional): A list of custom profanity words. Defaults to None.
|
|
25
27
|
|
|
26
28
|
Returns:
|
|
27
|
-
list: A list of profanity words
|
|
29
|
+
list: A list of profanity words.
|
|
30
|
+
|
|
31
|
+
Raises:
|
|
32
|
+
ValueError: If language is None and custom_words_list is None or empty.
|
|
28
33
|
"""
|
|
29
34
|
# Profanity words as a string
|
|
30
35
|
PROFANITY_WORDS = """
|
|
@@ -2740,35 +2745,102 @@ def load_profanity_words(language='English'):
|
|
|
2740
2745
|
if current_language:
|
|
2741
2746
|
profanity_lists[current_language] = current_words
|
|
2742
2747
|
|
|
2743
|
-
|
|
2744
|
-
|
|
2748
|
+
loaded_words = []
|
|
2749
|
+
if language is None:
|
|
2750
|
+
if not custom_words_list:
|
|
2751
|
+
raise ValueError("If language is None, custom_words_list must be provided and non-empty.")
|
|
2752
|
+
loaded_words = list(custom_words_list) # Use a copy
|
|
2753
|
+
elif language == 'All':
|
|
2745
2754
|
for words in profanity_lists.values():
|
|
2746
|
-
|
|
2747
|
-
|
|
2755
|
+
loaded_words.extend(words)
|
|
2756
|
+
if custom_words_list:
|
|
2757
|
+
loaded_words.extend(w for w in custom_words_list if w not in loaded_words)
|
|
2748
2758
|
else:
|
|
2749
|
-
|
|
2759
|
+
loaded_words = list(profanity_lists.get(language, [])) # Use a copy
|
|
2760
|
+
if custom_words_list:
|
|
2761
|
+
loaded_words.extend(w for w in custom_words_list if w not in loaded_words)
|
|
2762
|
+
|
|
2763
|
+
return list(set(loaded_words)) # Return unique words
|
|
2750
2764
|
|
|
2751
2765
|
|
|
2752
|
-
def
|
|
2766
|
+
def load_custom_profanity_from_file(filepath: str) -> list[str]:
|
|
2767
|
+
"""
|
|
2768
|
+
Loads a custom list of profanity words from a file.
|
|
2769
|
+
|
|
2770
|
+
The file should contain one word per line. Lines starting with '#' will be ignored as comments.
|
|
2771
|
+
Empty lines or lines with only whitespace will also be ignored.
|
|
2772
|
+
|
|
2773
|
+
Args:
|
|
2774
|
+
filepath (str): The path to the file containing custom profanity words.
|
|
2775
|
+
|
|
2776
|
+
Returns:
|
|
2777
|
+
list[str]: A list of words loaded from the file.
|
|
2778
|
+
|
|
2779
|
+
Raises:
|
|
2780
|
+
FileNotFoundError: If the specified filepath does not exist.
|
|
2781
|
+
IOError: For other file reading issues.
|
|
2782
|
+
"""
|
|
2783
|
+
custom_words = []
|
|
2784
|
+
try:
|
|
2785
|
+
with open(filepath, 'r', encoding='utf-8') as f:
|
|
2786
|
+
for line in f:
|
|
2787
|
+
word = line.strip()
|
|
2788
|
+
if word and not word.startswith('#'):
|
|
2789
|
+
custom_words.append(word)
|
|
2790
|
+
except FileNotFoundError:
|
|
2791
|
+
# Re-raise FileNotFoundError to be explicit, or handle as per specific project policy
|
|
2792
|
+
# For example, could log a warning and return empty list:
|
|
2793
|
+
# print(f"Warning: File not found at {filepath}. Returning empty custom word list.")
|
|
2794
|
+
# return []
|
|
2795
|
+
raise
|
|
2796
|
+
except IOError as e:
|
|
2797
|
+
# Handle other IO errors, perhaps log them or raise a more generic error
|
|
2798
|
+
# print(f"Warning: Error reading file {filepath}: {e}. Returning empty custom word list.")
|
|
2799
|
+
# return []
|
|
2800
|
+
raise IOError(f"Error reading custom profanity file {filepath}: {e}")
|
|
2801
|
+
return custom_words
|
|
2802
|
+
|
|
2803
|
+
|
|
2804
|
+
def detect_profanity(text_data, language='English', custom_words_list: list[str] = None):
|
|
2753
2805
|
"""
|
|
2754
2806
|
Detect profanity in text data using regex.
|
|
2755
2807
|
|
|
2756
2808
|
Args:
|
|
2757
2809
|
text_data (list): A list of strings representing the text data to analyze.
|
|
2758
|
-
language (str): The language used to detect profanity. Defaults to 'English'.
|
|
2810
|
+
language (str, optional): The language used to detect profanity. Defaults to 'English'.
|
|
2811
|
+
Can be None if custom_words_list is provided.
|
|
2812
|
+
Available languages include: All, Arabic, Czech, Danish, German,
|
|
2813
|
+
English, Esperanto, Persian, Finnish, Filipino, French,
|
|
2814
|
+
French (CA), Hindi, Hungarian, Italian, Japanese, Kabyle,
|
|
2815
|
+
Korean, Dutch, Norwegian, Polish, Portuguese, Russian,
|
|
2816
|
+
Swedish, Thai, Klingon, Turkish, Chinese.
|
|
2817
|
+
custom_words_list (list[str], optional): A list of custom profanity words. Defaults to None.
|
|
2759
2818
|
Returns:
|
|
2760
2819
|
list: A list of dictionaries where each dictionary represents a detected instance of profanity.
|
|
2761
2820
|
Each dictionary contains the following keys:
|
|
2762
2821
|
- "Line" (int): The line number where the profanity was detected.
|
|
2763
2822
|
- "Column" (int): The column number (position in the line) where the profanity starts.
|
|
2764
2823
|
- "Word" (str): The detected profanity word.
|
|
2765
|
-
- "Language" (str): The language
|
|
2824
|
+
- "Language" (str): The language string indicating the source of the profanity list.
|
|
2766
2825
|
"""
|
|
2767
|
-
profanity_keywords = load_profanity_words(language)
|
|
2826
|
+
profanity_keywords = load_profanity_words(language, custom_words_list)
|
|
2768
2827
|
detected = set() # Use a set to store unique occurrences
|
|
2769
2828
|
|
|
2829
|
+
display_language = "Custom"
|
|
2830
|
+
if language:
|
|
2831
|
+
if custom_words_list:
|
|
2832
|
+
display_language = f"Custom + {language}"
|
|
2833
|
+
else:
|
|
2834
|
+
display_language = language
|
|
2835
|
+
elif not custom_words_list:
|
|
2836
|
+
# This case should ideally be prevented by load_profanity_words raising an error,
|
|
2837
|
+
# but as a fallback, if somehow load_profanity_words returns empty and language is None
|
|
2838
|
+
# and no custom_words_list, we avoid error here.
|
|
2839
|
+
return []
|
|
2840
|
+
|
|
2841
|
+
|
|
2770
2842
|
for i, line in enumerate(text_data):
|
|
2771
|
-
# Skip lines with language markers
|
|
2843
|
+
# Skip lines with language markers (though less relevant if custom lists are primary)
|
|
2772
2844
|
if line.startswith("$") and line.endswith("$"):
|
|
2773
2845
|
continue
|
|
2774
2846
|
|
|
@@ -2776,36 +2848,39 @@ def detect_profanity(text_data, language='English'):
|
|
|
2776
2848
|
for profanity in profanity_keywords:
|
|
2777
2849
|
matches = re.finditer(r'\b{}\b'.format(re.escape(profanity)), line, flags=re.IGNORECASE)
|
|
2778
2850
|
for match in matches:
|
|
2779
|
-
|
|
2851
|
+
# Store with display_language, not the input `language` parameter
|
|
2852
|
+
detected.add((i + 1, match.start() + 1, display_language, profanity))
|
|
2780
2853
|
|
|
2781
2854
|
detections = [] # Initialize detections as a list
|
|
2782
2855
|
|
|
2783
2856
|
if detected:
|
|
2784
|
-
for line_num, col_num,
|
|
2857
|
+
for line_num, col_num, lang_display_name, word in detected:
|
|
2785
2858
|
detection_info = {
|
|
2786
2859
|
"Line": line_num,
|
|
2787
2860
|
"Column": col_num,
|
|
2788
2861
|
"Word": word,
|
|
2789
|
-
"Language":
|
|
2862
|
+
"Language": lang_display_name
|
|
2790
2863
|
}
|
|
2791
|
-
detections.append(detection_info)
|
|
2864
|
+
detections.append(detection_info)
|
|
2792
2865
|
|
|
2793
2866
|
return detections
|
|
2794
2867
|
|
|
2795
2868
|
|
|
2796
|
-
def remove_profanity(text_data, output_file=None, language='English'):
|
|
2869
|
+
def remove_profanity(text_data, output_file=None, language='English', custom_words_list: list[str] = None):
|
|
2797
2870
|
"""
|
|
2798
2871
|
Remove profanity from text data.
|
|
2799
2872
|
|
|
2800
2873
|
Args:
|
|
2801
2874
|
text_data (list): A list of strings representing the text data to clean.
|
|
2802
|
-
output_file (str): The file path to write the cleaned data. If None, cleaned data is not written to a file.
|
|
2803
|
-
language (str): The language for which to remove profanity. Defaults to 'English'.
|
|
2875
|
+
output_file (str, optional): The file path to write the cleaned data. If None, cleaned data is not written to a file.
|
|
2876
|
+
language (str, optional): The language for which to remove profanity. Defaults to 'English'.
|
|
2877
|
+
Can be None if custom_words_list is provided.
|
|
2878
|
+
custom_words_list (list[str], optional): A list of custom profanity words. Defaults to None.
|
|
2804
2879
|
|
|
2805
2880
|
Returns:
|
|
2806
2881
|
list: A list of strings representing the cleaned text data.
|
|
2807
2882
|
"""
|
|
2808
|
-
profanity_keywords = load_profanity_words(language)
|
|
2883
|
+
profanity_keywords = load_profanity_words(language, custom_words_list)
|
|
2809
2884
|
cleaned_data = []
|
|
2810
2885
|
|
|
2811
2886
|
for line in text_data:
|
|
@@ -1,6 +1,6 @@
|
|
|
1
|
-
Metadata-Version: 2.
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
2
|
Name: valx
|
|
3
|
-
Version: 0.2.
|
|
3
|
+
Version: 0.2.5
|
|
4
4
|
Summary: An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.
|
|
5
5
|
Home-page: https://github.com/infinitode/valx
|
|
6
6
|
Author: Infinitode Pty Ltd
|
|
@@ -19,6 +19,16 @@ Requires-Python: >=3.6
|
|
|
19
19
|
Description-Content-Type: text/markdown
|
|
20
20
|
License-File: LICENSE
|
|
21
21
|
Requires-Dist: scikit-learn
|
|
22
|
+
Dynamic: author
|
|
23
|
+
Dynamic: author-email
|
|
24
|
+
Dynamic: classifier
|
|
25
|
+
Dynamic: description
|
|
26
|
+
Dynamic: description-content-type
|
|
27
|
+
Dynamic: home-page
|
|
28
|
+
Dynamic: license-file
|
|
29
|
+
Dynamic: requires-dist
|
|
30
|
+
Dynamic: requires-python
|
|
31
|
+
Dynamic: summary
|
|
22
32
|
|
|
23
33
|
# ValX
|
|
24
34
|

|
|
@@ -30,9 +40,29 @@ Requires-Dist: scikit-learn
|
|
|
30
40
|
An open-source Python library for data cleaning tasks. It includes functions for profanity detection, and removal, and detection and removal of personal information. Also includes hate speech and offensive language detection and removal, using AI.
|
|
31
41
|
|
|
32
42
|
> [!IMPORTANT]
|
|
43
|
+
> If you are using `scikit-learn` versions older than version `1.3.0`, please also downgrade your version of `numpy` as stated below. Otherwise, you can continue to use your preferred version of `scikit-learn` without downgrading `numpy`.
|
|
44
|
+
>
|
|
33
45
|
> Please downgrade to `numpy` version `1.26.4`. Our ValX **DecisionTreeClassifier** AI model, relies on lower versions of `numpy`, because it was trained on these versions.
|
|
34
46
|
> For more information see: https://techoverflow.net/2024/07/23/how-to-fix-numpy-dtype-size-changed-may-indicate-binary-incompatibility-expected-96-from-c-header-got-88-from-pyobject/
|
|
35
47
|
|
|
48
|
+
> [!NOTE]
|
|
49
|
+
> ValX will automatically install a version of `scikit-learn` that is compatible with your device if you don't have one already.
|
|
50
|
+
|
|
51
|
+
## Changes in 0.2.5
|
|
52
|
+
|
|
53
|
+
ValX v0.2.5 introduces enhanced flexibility for profanity filtering by adding support for custom profanity lists:
|
|
54
|
+
|
|
55
|
+
- **Custom Profanity Word Lists**: Users can now provide their own lists of profane words directly as Python lists to the `detect_profanity` and `remove_profanity` functions via the new `custom_words_list` parameter.
|
|
56
|
+
- **Standalone Custom Lists**: Utilize your custom profanity list exclusively by setting the `language` parameter to `None`. ValX will then only use the words provided in `custom_words_list`.
|
|
57
|
+
- **Combined Lists**: Use a custom list in conjunction with ValX's built-in language-specific wordlists. Simply provide both a `language` (e.g., "English") and your `custom_words_list`. ValX will use the combined set of words.
|
|
58
|
+
- **Loading Custom Lists from File**: A new helper function, `load_custom_profanity_from_file(filepath)`, allows you to easily load custom profanity words from a text file.
|
|
59
|
+
- **File Format**: The file should contain one profanity word per line.
|
|
60
|
+
- Lines starting with a hash symbol (`#`) are treated as comments and ignored.
|
|
61
|
+
- Empty lines or lines containing only whitespace are also ignored.
|
|
62
|
+
- **Updated Detection Reporting**: The `detect_profanity` function's output now specifies the source of detected profanity more clearly (e.g., "Custom", "Custom + English").
|
|
63
|
+
|
|
64
|
+
These features give users greater control over the profanity filtering process, allowing for more tailored and specific use cases.
|
|
65
|
+
|
|
36
66
|
## Changes in 0.2.4
|
|
37
67
|
|
|
38
68
|
Fixed a major incompatibility issue with `scikit-learn` due to version changes in `scikit-learn v1.3.0` which causes compatibility issues with versions later than `1.2.2`. ValX can now be used with `scikit-learn` versions earlier and later than `1.3.0`!
|
|
@@ -130,25 +160,129 @@ Below is a complete list of all the available supported languages for ValX's pro
|
|
|
130
160
|
|
|
131
161
|
## Usage
|
|
132
162
|
|
|
133
|
-
###
|
|
163
|
+
### Profanity Detection and Removal
|
|
164
|
+
|
|
165
|
+
ValX allows for flexible profanity filtering using built-in language lists, custom word lists (provided as Python lists or loaded from files), or a combination of both.
|
|
166
|
+
|
|
167
|
+
**1. Basic Profanity Detection (Built-in Language)**
|
|
134
168
|
|
|
135
169
|
```python
|
|
136
170
|
from valx import detect_profanity
|
|
137
171
|
|
|
138
|
-
|
|
172
|
+
sample_text = ["This is some fuck and porn text."]
|
|
173
|
+
# Detect profanity using the English list
|
|
139
174
|
results = detect_profanity(sample_text, language='English')
|
|
140
|
-
|
|
175
|
+
# results will be:
|
|
176
|
+
# [
|
|
177
|
+
# {'Line': 1, 'Column': 14, 'Word': 'fuck', 'Language': 'English'},
|
|
178
|
+
# {'Line': 1, 'Column': 23, 'Word': 'porn', 'Language': 'English'}
|
|
179
|
+
# ]
|
|
180
|
+
print(results)
|
|
181
|
+
```
|
|
182
|
+
|
|
183
|
+
**2. Profanity Detection with a Custom Word List (Python List)**
|
|
184
|
+
|
|
185
|
+
You can provide your own list of words to filter.
|
|
186
|
+
|
|
187
|
+
```python
|
|
188
|
+
from valx import detect_profanity
|
|
189
|
+
|
|
190
|
+
sample_text = ["This contains custombadword1 and also asshole from English list."]
|
|
191
|
+
my_custom_words = ["custombadword1", "anothercustom"]
|
|
192
|
+
|
|
193
|
+
# Option A: Custom list ONLY (language=None)
|
|
194
|
+
results_custom_only = detect_profanity(sample_text, language=None, custom_words_list=my_custom_words)
|
|
195
|
+
# results_custom_only will detect "custombadword1" with Language: "Custom"
|
|
196
|
+
# [{'Line': 1, 'Column': 15, 'Word': 'custombadword1', 'Language': 'Custom'}]
|
|
197
|
+
print(results_custom_only)
|
|
198
|
+
|
|
199
|
+
# Option B: Custom list COMBINED with a built-in language
|
|
200
|
+
results_custom_plus_english = detect_profanity(sample_text, language="English", custom_words_list=my_custom_words)
|
|
201
|
+
# results_custom_plus_english will detect "custombadword1" and "asshole"
|
|
202
|
+
# Language will be "Custom + English"
|
|
203
|
+
# [
|
|
204
|
+
# {'Line': 1, 'Column': 15, 'Word': 'custombadword1', 'Language': 'Custom + English'},
|
|
205
|
+
# {'Line': 1, 'Column': 43, 'Word': 'asshole', 'Language': 'Custom + English'}
|
|
206
|
+
# ]
|
|
207
|
+
print(results_custom_plus_english)
|
|
141
208
|
```
|
|
142
209
|
|
|
143
|
-
|
|
210
|
+
**3. Loading Custom Profanity Words from a File**
|
|
211
|
+
|
|
212
|
+
ValX provides a helper function to load words from a text file (one word per line, '#' for comments).
|
|
144
213
|
|
|
145
214
|
```python
|
|
146
|
-
from valx import
|
|
215
|
+
from valx import detect_profanity, load_custom_profanity_from_file
|
|
216
|
+
|
|
217
|
+
# Assume 'my_profanity_file.txt' contains:
|
|
218
|
+
# customfileword1
|
|
219
|
+
# # this is a comment
|
|
220
|
+
# customfileword2
|
|
221
|
+
|
|
222
|
+
custom_words_from_file = load_custom_profanity_from_file("my_profanity_file.txt")
|
|
223
|
+
# custom_words_from_file will be: ['customfileword1', 'customfileword2']
|
|
224
|
+
|
|
225
|
+
sample_text_for_file = ["Text with customfileword1 and built-in shit."]
|
|
226
|
+
|
|
227
|
+
# Use file-loaded list with English built-in list
|
|
228
|
+
results_file_plus_english = detect_profanity(
|
|
229
|
+
sample_text_for_file,
|
|
230
|
+
language="English",
|
|
231
|
+
custom_words_list=custom_words_from_file
|
|
232
|
+
)
|
|
233
|
+
# Detects "customfileword1" and "shit", Language: "Custom + English"
|
|
234
|
+
print(results_file_plus_english)
|
|
235
|
+
|
|
236
|
+
# Use file-loaded list ONLY
|
|
237
|
+
results_file_only = detect_profanity(
|
|
238
|
+
sample_text_for_file,
|
|
239
|
+
language=None, # Important: set language to None
|
|
240
|
+
custom_words_list=custom_words_from_file
|
|
241
|
+
)
|
|
242
|
+
# Detects only "customfileword1", Language: "Custom"
|
|
243
|
+
print(results_file_only)
|
|
244
|
+
```
|
|
245
|
+
|
|
246
|
+
**Output Format for `detect_profanity`**
|
|
147
247
|
|
|
148
|
-
|
|
149
|
-
|
|
248
|
+
The `detect_profanity` function returns a list of dictionaries. Each dictionary includes:
|
|
249
|
+
- `"Line"`: The line number (1-indexed).
|
|
250
|
+
- `"Column"`: The column number (1-indexed) where the profanity starts.
|
|
251
|
+
- `"Word"`: The detected profanity word.
|
|
252
|
+
- `"Language"`: Indicates the source of the word list:
|
|
253
|
+
- `<LanguageName>` (e.g., "English"): If only a built-in language list was used.
|
|
254
|
+
- `"Custom"`: If `language=None` and only a `custom_words_list` was used.
|
|
255
|
+
- `"Custom + <LanguageName>"` (e.g., "Custom + English"): If both a built-in list and `custom_words_list` were used.
|
|
256
|
+
- `"Custom + All"`: If `language='All'` and `custom_words_list` were used.
|
|
257
|
+
|
|
258
|
+
|
|
259
|
+
**4. Removing Profanity**
|
|
260
|
+
|
|
261
|
+
`remove_profanity` works similarly, accepting `language` and `custom_words_list` parameters.
|
|
262
|
+
|
|
263
|
+
```python
|
|
264
|
+
from valx import remove_profanity, load_custom_profanity_from_file
|
|
265
|
+
|
|
266
|
+
sample_text = ["This is fuck, custombadword1, and text with customfileword1."]
|
|
267
|
+
my_custom_words = ["custombadword1"]
|
|
268
|
+
custom_words_from_file = load_custom_profanity_from_file("my_profanity_file.txt") # Assuming it contains 'customfileword1'
|
|
269
|
+
|
|
270
|
+
# Remove profanity using English built-in + my_custom_words + custom_words_from_file
|
|
271
|
+
all_custom_words = list(set(my_custom_words + custom_words_from_file)) # Combine and unique
|
|
272
|
+
|
|
273
|
+
cleaned_text = remove_profanity(
|
|
274
|
+
sample_text,
|
|
275
|
+
output_file="cleaned_output.txt", # Optional: saves to file
|
|
276
|
+
language="English",
|
|
277
|
+
custom_words_list=all_custom_words
|
|
278
|
+
)
|
|
279
|
+
# cleaned_text will have "fuck", "custombadword1", and "customfileword1" replaced with "bad word".
|
|
280
|
+
# e.g., ["This is bad word, bad word, and text with bad word."]
|
|
281
|
+
print(cleaned_text)
|
|
150
282
|
```
|
|
151
283
|
|
|
284
|
+
The `load_profanity_words` function (used internally) also accepts `language` and `custom_words_list` if you need direct access to the word lists.
|
|
285
|
+
|
|
152
286
|
### Detect Sensitive Information
|
|
153
287
|
|
|
154
288
|
```python
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|