valx 0.2.4__tar.gz → 0.2.6__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {valx-0.2.4/valx.egg-info → valx-0.2.6}/PKG-INFO +187 -10
- valx-0.2.6/README.md +357 -0
- {valx-0.2.4 → valx-0.2.6}/setup.py +1 -1
- {valx-0.2.4 → valx-0.2.6}/valx/__init__.py +318 -240
- {valx-0.2.4 → valx-0.2.6/valx.egg-info}/PKG-INFO +187 -10
- valx-0.2.4/README.md +0 -190
- {valx-0.2.4 → valx-0.2.6}/LICENSE +0 -0
- {valx-0.2.4 → valx-0.2.6}/setup.cfg +0 -0
- {valx-0.2.4 → valx-0.2.6}/valx/models/classifier_upgraded.pkl +0 -0
- {valx-0.2.4 → valx-0.2.6}/valx/models/count_vectorizer.sav +0 -0
- {valx-0.2.4 → valx-0.2.6}/valx/models/decision_tree_model.sav +0 -0
- {valx-0.2.4 → valx-0.2.6}/valx.egg-info/SOURCES.txt +0 -0
- {valx-0.2.4 → valx-0.2.6}/valx.egg-info/dependency_links.txt +0 -0
- {valx-0.2.4 → valx-0.2.6}/valx.egg-info/requires.txt +0 -0
- {valx-0.2.4 → valx-0.2.6}/valx.egg-info/top_level.txt +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
|
-
Metadata-Version: 2.
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
2
|
Name: valx
|
|
3
|
-
Version: 0.2.
|
|
3
|
+
Version: 0.2.6
|
|
4
4
|
Summary: An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.
|
|
5
5
|
Home-page: https://github.com/infinitode/valx
|
|
6
6
|
Author: Infinitode Pty Ltd
|
|
@@ -19,8 +19,19 @@ Requires-Python: >=3.6
|
|
|
19
19
|
Description-Content-Type: text/markdown
|
|
20
20
|
License-File: LICENSE
|
|
21
21
|
Requires-Dist: scikit-learn
|
|
22
|
+
Dynamic: author
|
|
23
|
+
Dynamic: author-email
|
|
24
|
+
Dynamic: classifier
|
|
25
|
+
Dynamic: description
|
|
26
|
+
Dynamic: description-content-type
|
|
27
|
+
Dynamic: home-page
|
|
28
|
+
Dynamic: license-file
|
|
29
|
+
Dynamic: requires-dist
|
|
30
|
+
Dynamic: requires-python
|
|
31
|
+
Dynamic: summary
|
|
22
32
|
|
|
23
33
|
# ValX
|
|
34
|
+
|
|
24
35
|

|
|
25
36
|
[](https://github.com/infinitode/valx)
|
|
26
37
|

|
|
@@ -30,9 +41,37 @@ Requires-Dist: scikit-learn
|
|
|
30
41
|
An open-source Python library for data cleaning tasks. It includes functions for profanity detection, and removal, and detection and removal of personal information. Also includes hate speech and offensive language detection and removal, using AI.
|
|
31
42
|
|
|
32
43
|
> [!IMPORTANT]
|
|
44
|
+
> If you are using `scikit-learn` versions older than version `1.3.0`, please also downgrade your version of `numpy` as stated below. Otherwise, you can continue to use your preferred version of `scikit-learn` without downgrading `numpy`.
|
|
45
|
+
>
|
|
33
46
|
> Please downgrade to `numpy` version `1.26.4`. Our ValX **DecisionTreeClassifier** AI model, relies on lower versions of `numpy`, because it was trained on these versions.
|
|
34
47
|
> For more information see: https://techoverflow.net/2024/07/23/how-to-fix-numpy-dtype-size-changed-may-indicate-binary-incompatibility-expected-96-from-c-header-got-88-from-pyobject/
|
|
35
48
|
|
|
49
|
+
> [!NOTE]
|
|
50
|
+
> ValX will automatically install a version of `scikit-learn` that is compatible with your device if you don't have one already.
|
|
51
|
+
|
|
52
|
+
## Changes in 0.2.6
|
|
53
|
+
|
|
54
|
+
ValX v0.2.6 fixes a major bug, view the issue here: https://github.com/Infinitode/ValX/issues/4, where profanity lists for multiple languages were missing, under wrong languages, or simply incomplete.
|
|
55
|
+
|
|
56
|
+
Version 0.2.6 introduces fixes for this using the original language lists data, and includes new handling for languages, including:
|
|
57
|
+
|
|
58
|
+
- Case insensitivity for language selection: "English", "EN", "en", or variants like "enGliSh" will all work for language selection in the `detect_profanity` and `remove_profanity` functions.
|
|
59
|
+
|
|
60
|
+
## Changes in 0.2.5
|
|
61
|
+
|
|
62
|
+
ValX v0.2.5 introduces enhanced flexibility for profanity filtering by adding support for custom profanity lists:
|
|
63
|
+
|
|
64
|
+
- **Custom Profanity Word Lists**: Users can now provide their own lists of profane words directly as Python lists to the `detect_profanity` and `remove_profanity` functions via the new `custom_words_list` parameter.
|
|
65
|
+
- **Standalone Custom Lists**: Utilize your custom profanity list exclusively by setting the `language` parameter to `None`. ValX will then only use the words provided in `custom_words_list`.
|
|
66
|
+
- **Combined Lists**: Use a custom list in conjunction with ValX's built-in language-specific wordlists. Simply provide both a `language` (e.g., "English") and your `custom_words_list`. ValX will use the combined set of words.
|
|
67
|
+
- **Loading Custom Lists from File**: A new helper function, `load_custom_profanity_from_file(filepath)`, allows you to easily load custom profanity words from a text file.
|
|
68
|
+
- **File Format**: The file should contain one profanity word per line.
|
|
69
|
+
- Lines starting with a hash symbol (`#`) are treated as comments and ignored.
|
|
70
|
+
- Empty lines or lines containing only whitespace are also ignored.
|
|
71
|
+
- **Updated Detection Reporting**: The `detect_profanity` function's output now specifies the source of detected profanity more clearly (e.g., "Custom", "Custom + English").
|
|
72
|
+
|
|
73
|
+
These features give users greater control over the profanity filtering process, allowing for more tailored and specific use cases.
|
|
74
|
+
|
|
36
75
|
## Changes in 0.2.4
|
|
37
76
|
|
|
38
77
|
Fixed a major incompatibility issue with `scikit-learn` due to version changes in `scikit-learn v1.3.0` which causes compatibility issues with versions later than `1.2.2`. ValX can now be used with `scikit-learn` versions earlier and later than `1.3.0`!
|
|
@@ -44,6 +83,7 @@ We've also removed `scikit-learn==1.2.2` as a dependency, as most versions of `s
|
|
|
44
83
|
We have introduced a new optional `info_type` parameter into our `detect_sensitive_information`, and `remove_sensitive_information` functions, to allow you to have fine-grained control over what sensitive information you want to detect or remove.
|
|
45
84
|
|
|
46
85
|
Also introduced more detection patterns for other types of sensitive information, including:
|
|
86
|
+
|
|
47
87
|
- `"iban"`: International Bank Account Number.
|
|
48
88
|
- `"mrn"`: Medical Record Number (may not work correctly, depending on provider and country).
|
|
49
89
|
- `"icd10"`: International Classification of Diseases, Tenth Revision.
|
|
@@ -56,6 +96,7 @@ Also introduced more detection patterns for other types of sensitive information
|
|
|
56
96
|
## Changes in 0.2.2
|
|
57
97
|
|
|
58
98
|
We have refactored and changed the `detect_profanity` function:
|
|
99
|
+
|
|
59
100
|
- Removed unnecessary printing
|
|
60
101
|
- Now returns more information about each found profanity, including `Line`, `Column`, `Word`, and `Language`.
|
|
61
102
|
|
|
@@ -97,58 +138,191 @@ Please ensure that you have one of these Python versions installed before using
|
|
|
97
138
|
- **Remove Hate Speech**: Remove hate speech or offensive speech in text, using AI.
|
|
98
139
|
|
|
99
140
|
### List of supported languages for profanity detection and removal
|
|
141
|
+
|
|
100
142
|
Below is a complete list of all the available supported languages for ValX's profanity detection and removal functions which are valid values for `language`:
|
|
101
143
|
|
|
102
|
-
-
|
|
144
|
+
- All
|
|
103
145
|
- Arabic
|
|
146
|
+
- AR
|
|
104
147
|
- Czech
|
|
148
|
+
- CS
|
|
105
149
|
- Danish
|
|
150
|
+
- DA
|
|
106
151
|
- German
|
|
152
|
+
- DE
|
|
107
153
|
- English
|
|
154
|
+
- EN
|
|
108
155
|
- Esperanto
|
|
156
|
+
- EO
|
|
109
157
|
- Persian
|
|
110
158
|
- Finnish
|
|
159
|
+
- FI
|
|
111
160
|
- Filipino
|
|
161
|
+
- FIL
|
|
112
162
|
- French
|
|
163
|
+
- FR
|
|
113
164
|
- French (CA)
|
|
165
|
+
- FR-CA-U-SD-CAQC
|
|
114
166
|
- Hindi
|
|
167
|
+
- HI
|
|
115
168
|
- Hungarian
|
|
169
|
+
- HU
|
|
116
170
|
- Italian
|
|
171
|
+
- IT
|
|
117
172
|
- Japanese
|
|
173
|
+
- JA
|
|
118
174
|
- Kabyle
|
|
175
|
+
- KAB
|
|
119
176
|
- Korean
|
|
177
|
+
- KO
|
|
120
178
|
- Dutch
|
|
179
|
+
- NL
|
|
121
180
|
- Norwegian
|
|
181
|
+
- NO
|
|
122
182
|
- Polish
|
|
183
|
+
- PL
|
|
123
184
|
- Portuguese
|
|
185
|
+
- PT
|
|
124
186
|
- Russian
|
|
187
|
+
- RU
|
|
188
|
+
- Spanish
|
|
189
|
+
- ES
|
|
125
190
|
- Swedish
|
|
191
|
+
- SV
|
|
126
192
|
- Thai
|
|
193
|
+
- TH
|
|
127
194
|
- Klingon
|
|
195
|
+
- TLH
|
|
128
196
|
- Turkish
|
|
197
|
+
- TR
|
|
129
198
|
- Chinese
|
|
199
|
+
- ZH
|
|
130
200
|
|
|
131
201
|
## Usage
|
|
132
202
|
|
|
133
|
-
###
|
|
203
|
+
### Profanity Detection and Removal
|
|
204
|
+
|
|
205
|
+
ValX allows for flexible profanity filtering using built-in language lists, custom word lists (provided as Python lists or loaded from files), or a combination of both.
|
|
206
|
+
|
|
207
|
+
**1. Basic Profanity Detection (Built-in Language)**
|
|
134
208
|
|
|
135
209
|
```python
|
|
136
210
|
from valx import detect_profanity
|
|
137
211
|
|
|
138
|
-
|
|
212
|
+
sample_text = ["This is some fuck and porn text."]
|
|
213
|
+
# Detect profanity using the English list
|
|
139
214
|
results = detect_profanity(sample_text, language='English')
|
|
140
|
-
|
|
215
|
+
# results will be:
|
|
216
|
+
# [
|
|
217
|
+
# {'Line': 1, 'Column': 14, 'Word': 'fuck', 'Language': 'English'},
|
|
218
|
+
# {'Line': 1, 'Column': 23, 'Word': 'porn', 'Language': 'English'}
|
|
219
|
+
# ]
|
|
220
|
+
print(results)
|
|
221
|
+
```
|
|
222
|
+
|
|
223
|
+
**2. Profanity Detection with a Custom Word List (Python List)**
|
|
224
|
+
|
|
225
|
+
You can provide your own list of words to filter.
|
|
226
|
+
|
|
227
|
+
```python
|
|
228
|
+
from valx import detect_profanity
|
|
229
|
+
|
|
230
|
+
sample_text = ["This contains custombadword1 and also asshole from English list."]
|
|
231
|
+
my_custom_words = ["custombadword1", "anothercustom"]
|
|
232
|
+
|
|
233
|
+
# Option A: Custom list ONLY (language=None)
|
|
234
|
+
results_custom_only = detect_profanity(sample_text, language=None, custom_words_list=my_custom_words)
|
|
235
|
+
# results_custom_only will detect "custombadword1" with Language: "Custom"
|
|
236
|
+
# [{'Line': 1, 'Column': 15, 'Word': 'custombadword1', 'Language': 'Custom'}]
|
|
237
|
+
print(results_custom_only)
|
|
238
|
+
|
|
239
|
+
# Option B: Custom list COMBINED with a built-in language
|
|
240
|
+
results_custom_plus_english = detect_profanity(sample_text, language="English", custom_words_list=my_custom_words)
|
|
241
|
+
# results_custom_plus_english will detect "custombadword1" and "asshole"
|
|
242
|
+
# Language will be "Custom + English"
|
|
243
|
+
# [
|
|
244
|
+
# {'Line': 1, 'Column': 15, 'Word': 'custombadword1', 'Language': 'Custom + English'},
|
|
245
|
+
# {'Line': 1, 'Column': 43, 'Word': 'asshole', 'Language': 'Custom + English'}
|
|
246
|
+
# ]
|
|
247
|
+
print(results_custom_plus_english)
|
|
141
248
|
```
|
|
142
249
|
|
|
143
|
-
|
|
250
|
+
**3. Loading Custom Profanity Words from a File**
|
|
251
|
+
|
|
252
|
+
ValX provides a helper function to load words from a text file (one word per line, '#' for comments).
|
|
144
253
|
|
|
145
254
|
```python
|
|
146
|
-
from valx import
|
|
255
|
+
from valx import detect_profanity, load_custom_profanity_from_file
|
|
256
|
+
|
|
257
|
+
# Assume 'my_profanity_file.txt' contains:
|
|
258
|
+
# customfileword1
|
|
259
|
+
# # this is a comment
|
|
260
|
+
# customfileword2
|
|
261
|
+
|
|
262
|
+
custom_words_from_file = load_custom_profanity_from_file("my_profanity_file.txt")
|
|
263
|
+
# custom_words_from_file will be: ['customfileword1', 'customfileword2']
|
|
264
|
+
|
|
265
|
+
sample_text_for_file = ["Text with customfileword1 and built-in shit."]
|
|
266
|
+
|
|
267
|
+
# Use file-loaded list with English built-in list
|
|
268
|
+
results_file_plus_english = detect_profanity(
|
|
269
|
+
sample_text_for_file,
|
|
270
|
+
language="English",
|
|
271
|
+
custom_words_list=custom_words_from_file
|
|
272
|
+
)
|
|
273
|
+
# Detects "customfileword1" and "shit", Language: "Custom + English"
|
|
274
|
+
print(results_file_plus_english)
|
|
275
|
+
|
|
276
|
+
# Use file-loaded list ONLY
|
|
277
|
+
results_file_only = detect_profanity(
|
|
278
|
+
sample_text_for_file,
|
|
279
|
+
language=None, # Important: set language to None
|
|
280
|
+
custom_words_list=custom_words_from_file
|
|
281
|
+
)
|
|
282
|
+
# Detects only "customfileword1", Language: "Custom"
|
|
283
|
+
print(results_file_only)
|
|
284
|
+
```
|
|
285
|
+
|
|
286
|
+
**Output Format for `detect_profanity`**
|
|
147
287
|
|
|
148
|
-
|
|
149
|
-
|
|
288
|
+
The `detect_profanity` function returns a list of dictionaries. Each dictionary includes:
|
|
289
|
+
|
|
290
|
+
- `"Line"`: The line number (1-indexed).
|
|
291
|
+
- `"Column"`: The column number (1-indexed) where the profanity starts.
|
|
292
|
+
- `"Word"`: The detected profanity word.
|
|
293
|
+
- `"Language"`: Indicates the source of the word list:
|
|
294
|
+
- `<LanguageName>` (e.g., "English"): If only a built-in language list was used.
|
|
295
|
+
- `"Custom"`: If `language=None` and only a `custom_words_list` was used.
|
|
296
|
+
- `"Custom + <LanguageName>"` (e.g., "Custom + English"): If both a built-in list and `custom_words_list` were used.
|
|
297
|
+
- `"Custom + All"`: If `language='All'` and `custom_words_list` were used.
|
|
298
|
+
|
|
299
|
+
**4. Removing Profanity**
|
|
300
|
+
|
|
301
|
+
`remove_profanity` works similarly, accepting `language` and `custom_words_list` parameters.
|
|
302
|
+
|
|
303
|
+
```python
|
|
304
|
+
from valx import remove_profanity, load_custom_profanity_from_file
|
|
305
|
+
|
|
306
|
+
sample_text = ["This is fuck, custombadword1, and text with customfileword1."]
|
|
307
|
+
my_custom_words = ["custombadword1"]
|
|
308
|
+
custom_words_from_file = load_custom_profanity_from_file("my_profanity_file.txt") # Assuming it contains 'customfileword1'
|
|
309
|
+
|
|
310
|
+
# Remove profanity using English built-in + my_custom_words + custom_words_from_file
|
|
311
|
+
all_custom_words = list(set(my_custom_words + custom_words_from_file)) # Combine and unique
|
|
312
|
+
|
|
313
|
+
cleaned_text = remove_profanity(
|
|
314
|
+
sample_text,
|
|
315
|
+
output_file="cleaned_output.txt", # Optional: saves to file
|
|
316
|
+
language="English",
|
|
317
|
+
custom_words_list=all_custom_words
|
|
318
|
+
)
|
|
319
|
+
# cleaned_text will have "fuck", "custombadword1", and "customfileword1" replaced with "bad word".
|
|
320
|
+
# e.g., ["This is bad word, bad word, and text with bad word."]
|
|
321
|
+
print(cleaned_text)
|
|
150
322
|
```
|
|
151
323
|
|
|
324
|
+
The `load_profanity_words` function (used internally) also accepts `language` and `custom_words_list` if you need direct access to the word lists.
|
|
325
|
+
|
|
152
326
|
### Detect Sensitive Information
|
|
153
327
|
|
|
154
328
|
```python
|
|
@@ -181,6 +355,7 @@ outcome_of_detection = detect_hate_speech("You are stupid.")
|
|
|
181
355
|
|
|
182
356
|
> [!IMPORTANT]
|
|
183
357
|
> The model's possible outputs are:
|
|
358
|
+
>
|
|
184
359
|
> - `['Hate Speech']`: The text was flagged and contained hate speech.
|
|
185
360
|
> - `['Offensive Speech']`: The text was flagged and contained offensive speech.
|
|
186
361
|
> - `['No Hate and Offensive Speech']`: The text was not flagged for any hate speech or offensive speech.
|
|
@@ -197,7 +372,9 @@ Contributions are welcome! If you encounter any issues, have suggestions, or wan
|
|
|
197
372
|
ValX is released under the terms of the **MIT License (Modified)**. Please see the [LICENSE](https://github.com/infinitode/valx/blob/main/LICENSE) file for the full text.
|
|
198
373
|
|
|
199
374
|
### Derived licenses
|
|
375
|
+
|
|
200
376
|
---
|
|
377
|
+
|
|
201
378
|
ValX uses data from this GitHub repository:
|
|
202
379
|
https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words/
|
|
203
380
|
© 2012-2020 Shutterstock, Inc.
|
valx-0.2.6/README.md
ADDED
|
@@ -0,0 +1,357 @@
|
|
|
1
|
+
# ValX
|
|
2
|
+
|
|
3
|
+

|
|
4
|
+
[](https://github.com/infinitode/valx)
|
|
5
|
+

|
|
6
|
+

|
|
7
|
+

|
|
8
|
+
|
|
9
|
+
An open-source Python library for data cleaning tasks. It includes functions for profanity detection, and removal, and detection and removal of personal information. Also includes hate speech and offensive language detection and removal, using AI.
|
|
10
|
+
|
|
11
|
+
> [!IMPORTANT]
|
|
12
|
+
> If you are using `scikit-learn` versions older than version `1.3.0`, please also downgrade your version of `numpy` as stated below. Otherwise, you can continue to use your preferred version of `scikit-learn` without downgrading `numpy`.
|
|
13
|
+
>
|
|
14
|
+
> Please downgrade to `numpy` version `1.26.4`. Our ValX **DecisionTreeClassifier** AI model, relies on lower versions of `numpy`, because it was trained on these versions.
|
|
15
|
+
> For more information see: https://techoverflow.net/2024/07/23/how-to-fix-numpy-dtype-size-changed-may-indicate-binary-incompatibility-expected-96-from-c-header-got-88-from-pyobject/
|
|
16
|
+
|
|
17
|
+
> [!NOTE]
|
|
18
|
+
> ValX will automatically install a version of `scikit-learn` that is compatible with your device if you don't have one already.
|
|
19
|
+
|
|
20
|
+
## Changes in 0.2.6
|
|
21
|
+
|
|
22
|
+
ValX v0.2.6 fixes a major bug, view the issue here: https://github.com/Infinitode/ValX/issues/4, where profanity lists for multiple languages were missing, under wrong languages, or simply incomplete.
|
|
23
|
+
|
|
24
|
+
Version 0.2.6 introduces fixes for this using the original language lists data, and includes new handling for languages, including:
|
|
25
|
+
|
|
26
|
+
- Case insensitivity for language selection: "English", "EN", "en", or variants like "enGliSh" will all work for language selection in the `detect_profanity` and `remove_profanity` functions.
|
|
27
|
+
|
|
28
|
+
## Changes in 0.2.5
|
|
29
|
+
|
|
30
|
+
ValX v0.2.5 introduces enhanced flexibility for profanity filtering by adding support for custom profanity lists:
|
|
31
|
+
|
|
32
|
+
- **Custom Profanity Word Lists**: Users can now provide their own lists of profane words directly as Python lists to the `detect_profanity` and `remove_profanity` functions via the new `custom_words_list` parameter.
|
|
33
|
+
- **Standalone Custom Lists**: Utilize your custom profanity list exclusively by setting the `language` parameter to `None`. ValX will then only use the words provided in `custom_words_list`.
|
|
34
|
+
- **Combined Lists**: Use a custom list in conjunction with ValX's built-in language-specific wordlists. Simply provide both a `language` (e.g., "English") and your `custom_words_list`. ValX will use the combined set of words.
|
|
35
|
+
- **Loading Custom Lists from File**: A new helper function, `load_custom_profanity_from_file(filepath)`, allows you to easily load custom profanity words from a text file.
|
|
36
|
+
- **File Format**: The file should contain one profanity word per line.
|
|
37
|
+
- Lines starting with a hash symbol (`#`) are treated as comments and ignored.
|
|
38
|
+
- Empty lines or lines containing only whitespace are also ignored.
|
|
39
|
+
- **Updated Detection Reporting**: The `detect_profanity` function's output now specifies the source of detected profanity more clearly (e.g., "Custom", "Custom + English").
|
|
40
|
+
|
|
41
|
+
These features give users greater control over the profanity filtering process, allowing for more tailored and specific use cases.
|
|
42
|
+
|
|
43
|
+
## Changes in 0.2.4
|
|
44
|
+
|
|
45
|
+
Fixed a major incompatibility issue with `scikit-learn` due to version changes in `scikit-learn v1.3.0` which causes compatibility issues with versions later than `1.2.2`. ValX can now be used with `scikit-learn` versions earlier and later than `1.3.0`!
|
|
46
|
+
|
|
47
|
+
We've also removed `scikit-learn==1.2.2` as a dependency, as most versions of `scikit-learn` will now work.
|
|
48
|
+
|
|
49
|
+
## Changes in 0.2.3
|
|
50
|
+
|
|
51
|
+
We have introduced a new optional `info_type` parameter into our `detect_sensitive_information`, and `remove_sensitive_information` functions, to allow you to have fine-grained control over what sensitive information you want to detect or remove.
|
|
52
|
+
|
|
53
|
+
Also introduced more detection patterns for other types of sensitive information, including:
|
|
54
|
+
|
|
55
|
+
- `"iban"`: International Bank Account Number.
|
|
56
|
+
- `"mrn"`: Medical Record Number (may not work correctly, depending on provider and country).
|
|
57
|
+
- `"icd10"`: International Classification of Diseases, Tenth Revision.
|
|
58
|
+
- `"geo_coords"`: Geo-coordinates (latitude and longitude in decimal degrees format).
|
|
59
|
+
- `"username"`: Username handles (@username).
|
|
60
|
+
- `"file_path"`: File paths (general patterns for both Windows and Unix paths).
|
|
61
|
+
- `"bitcoin_wallet"`: Cryptocurrency wallet address.
|
|
62
|
+
- `"ethereum_wallet"`: Cryptocurrency wallet addresses.
|
|
63
|
+
|
|
64
|
+
## Changes in 0.2.2
|
|
65
|
+
|
|
66
|
+
We have refactored and changed the `detect_profanity` function:
|
|
67
|
+
|
|
68
|
+
- Removed unnecessary printing
|
|
69
|
+
- Now returns more information about each found profanity, including `Line`, `Column`, `Word`, and `Language`.
|
|
70
|
+
|
|
71
|
+
> [!NOTE]
|
|
72
|
+
> You can view [ValX's package documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more information on changes.
|
|
73
|
+
|
|
74
|
+
## Changes in 0.2.1
|
|
75
|
+
|
|
76
|
+
Using the AI models in ValX, you can now automatically remove hate speech, or offensive speech from your text data, without needing to run detection and write your own custom implementation method.
|
|
77
|
+
|
|
78
|
+
## Installation
|
|
79
|
+
|
|
80
|
+
You can install ValX using pip:
|
|
81
|
+
|
|
82
|
+
```bash
|
|
83
|
+
pip install valx
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
## Supported Python Versions
|
|
87
|
+
|
|
88
|
+
ValX supports the following Python versions:
|
|
89
|
+
|
|
90
|
+
- Python 3.6
|
|
91
|
+
- Python 3.7
|
|
92
|
+
- Python 3.8
|
|
93
|
+
- Python 3.9
|
|
94
|
+
- Python 3.10
|
|
95
|
+
- Python 3.11/Later (Preferred)
|
|
96
|
+
|
|
97
|
+
Please ensure that you have one of these Python versions installed before using ValX. ValX may not work as expected on lower versions of Python than the supported.
|
|
98
|
+
|
|
99
|
+
## Features
|
|
100
|
+
|
|
101
|
+
- **Profanity Detection**: Detect profane and NSFW words or terms.
|
|
102
|
+
- **Remove Profanity**: Remove profane and NSFW words or terms.
|
|
103
|
+
- **Detect Sensitive Information**: Detect sensitive information in text data.
|
|
104
|
+
- **Remove Sensitive Information**: Remove sensitive information from text data.
|
|
105
|
+
- **Detect Hate Speech**: Detect hate speech or offensive speech in text, using AI.
|
|
106
|
+
- **Remove Hate Speech**: Remove hate speech or offensive speech in text, using AI.
|
|
107
|
+
|
|
108
|
+
### List of supported languages for profanity detection and removal
|
|
109
|
+
|
|
110
|
+
Below is a complete list of all the available supported languages for ValX's profanity detection and removal functions which are valid values for `language`:
|
|
111
|
+
|
|
112
|
+
- All
|
|
113
|
+
- Arabic
|
|
114
|
+
- AR
|
|
115
|
+
- Czech
|
|
116
|
+
- CS
|
|
117
|
+
- Danish
|
|
118
|
+
- DA
|
|
119
|
+
- German
|
|
120
|
+
- DE
|
|
121
|
+
- English
|
|
122
|
+
- EN
|
|
123
|
+
- Esperanto
|
|
124
|
+
- EO
|
|
125
|
+
- Persian
|
|
126
|
+
- Finnish
|
|
127
|
+
- FI
|
|
128
|
+
- Filipino
|
|
129
|
+
- FIL
|
|
130
|
+
- French
|
|
131
|
+
- FR
|
|
132
|
+
- French (CA)
|
|
133
|
+
- FR-CA-U-SD-CAQC
|
|
134
|
+
- Hindi
|
|
135
|
+
- HI
|
|
136
|
+
- Hungarian
|
|
137
|
+
- HU
|
|
138
|
+
- Italian
|
|
139
|
+
- IT
|
|
140
|
+
- Japanese
|
|
141
|
+
- JA
|
|
142
|
+
- Kabyle
|
|
143
|
+
- KAB
|
|
144
|
+
- Korean
|
|
145
|
+
- KO
|
|
146
|
+
- Dutch
|
|
147
|
+
- NL
|
|
148
|
+
- Norwegian
|
|
149
|
+
- NO
|
|
150
|
+
- Polish
|
|
151
|
+
- PL
|
|
152
|
+
- Portuguese
|
|
153
|
+
- PT
|
|
154
|
+
- Russian
|
|
155
|
+
- RU
|
|
156
|
+
- Spanish
|
|
157
|
+
- ES
|
|
158
|
+
- Swedish
|
|
159
|
+
- SV
|
|
160
|
+
- Thai
|
|
161
|
+
- TH
|
|
162
|
+
- Klingon
|
|
163
|
+
- TLH
|
|
164
|
+
- Turkish
|
|
165
|
+
- TR
|
|
166
|
+
- Chinese
|
|
167
|
+
- ZH
|
|
168
|
+
|
|
169
|
+
## Usage
|
|
170
|
+
|
|
171
|
+
### Profanity Detection and Removal
|
|
172
|
+
|
|
173
|
+
ValX allows for flexible profanity filtering using built-in language lists, custom word lists (provided as Python lists or loaded from files), or a combination of both.
|
|
174
|
+
|
|
175
|
+
**1. Basic Profanity Detection (Built-in Language)**
|
|
176
|
+
|
|
177
|
+
```python
|
|
178
|
+
from valx import detect_profanity
|
|
179
|
+
|
|
180
|
+
sample_text = ["This is some fuck and porn text."]
|
|
181
|
+
# Detect profanity using the English list
|
|
182
|
+
results = detect_profanity(sample_text, language='English')
|
|
183
|
+
# results will be:
|
|
184
|
+
# [
|
|
185
|
+
# {'Line': 1, 'Column': 14, 'Word': 'fuck', 'Language': 'English'},
|
|
186
|
+
# {'Line': 1, 'Column': 23, 'Word': 'porn', 'Language': 'English'}
|
|
187
|
+
# ]
|
|
188
|
+
print(results)
|
|
189
|
+
```
|
|
190
|
+
|
|
191
|
+
**2. Profanity Detection with a Custom Word List (Python List)**
|
|
192
|
+
|
|
193
|
+
You can provide your own list of words to filter.
|
|
194
|
+
|
|
195
|
+
```python
|
|
196
|
+
from valx import detect_profanity
|
|
197
|
+
|
|
198
|
+
sample_text = ["This contains custombadword1 and also asshole from English list."]
|
|
199
|
+
my_custom_words = ["custombadword1", "anothercustom"]
|
|
200
|
+
|
|
201
|
+
# Option A: Custom list ONLY (language=None)
|
|
202
|
+
results_custom_only = detect_profanity(sample_text, language=None, custom_words_list=my_custom_words)
|
|
203
|
+
# results_custom_only will detect "custombadword1" with Language: "Custom"
|
|
204
|
+
# [{'Line': 1, 'Column': 15, 'Word': 'custombadword1', 'Language': 'Custom'}]
|
|
205
|
+
print(results_custom_only)
|
|
206
|
+
|
|
207
|
+
# Option B: Custom list COMBINED with a built-in language
|
|
208
|
+
results_custom_plus_english = detect_profanity(sample_text, language="English", custom_words_list=my_custom_words)
|
|
209
|
+
# results_custom_plus_english will detect "custombadword1" and "asshole"
|
|
210
|
+
# Language will be "Custom + English"
|
|
211
|
+
# [
|
|
212
|
+
# {'Line': 1, 'Column': 15, 'Word': 'custombadword1', 'Language': 'Custom + English'},
|
|
213
|
+
# {'Line': 1, 'Column': 43, 'Word': 'asshole', 'Language': 'Custom + English'}
|
|
214
|
+
# ]
|
|
215
|
+
print(results_custom_plus_english)
|
|
216
|
+
```
|
|
217
|
+
|
|
218
|
+
**3. Loading Custom Profanity Words from a File**
|
|
219
|
+
|
|
220
|
+
ValX provides a helper function to load words from a text file (one word per line, '#' for comments).
|
|
221
|
+
|
|
222
|
+
```python
|
|
223
|
+
from valx import detect_profanity, load_custom_profanity_from_file
|
|
224
|
+
|
|
225
|
+
# Assume 'my_profanity_file.txt' contains:
|
|
226
|
+
# customfileword1
|
|
227
|
+
# # this is a comment
|
|
228
|
+
# customfileword2
|
|
229
|
+
|
|
230
|
+
custom_words_from_file = load_custom_profanity_from_file("my_profanity_file.txt")
|
|
231
|
+
# custom_words_from_file will be: ['customfileword1', 'customfileword2']
|
|
232
|
+
|
|
233
|
+
sample_text_for_file = ["Text with customfileword1 and built-in shit."]
|
|
234
|
+
|
|
235
|
+
# Use file-loaded list with English built-in list
|
|
236
|
+
results_file_plus_english = detect_profanity(
|
|
237
|
+
sample_text_for_file,
|
|
238
|
+
language="English",
|
|
239
|
+
custom_words_list=custom_words_from_file
|
|
240
|
+
)
|
|
241
|
+
# Detects "customfileword1" and "shit", Language: "Custom + English"
|
|
242
|
+
print(results_file_plus_english)
|
|
243
|
+
|
|
244
|
+
# Use file-loaded list ONLY
|
|
245
|
+
results_file_only = detect_profanity(
|
|
246
|
+
sample_text_for_file,
|
|
247
|
+
language=None, # Important: set language to None
|
|
248
|
+
custom_words_list=custom_words_from_file
|
|
249
|
+
)
|
|
250
|
+
# Detects only "customfileword1", Language: "Custom"
|
|
251
|
+
print(results_file_only)
|
|
252
|
+
```
|
|
253
|
+
|
|
254
|
+
**Output Format for `detect_profanity`**
|
|
255
|
+
|
|
256
|
+
The `detect_profanity` function returns a list of dictionaries. Each dictionary includes:
|
|
257
|
+
|
|
258
|
+
- `"Line"`: The line number (1-indexed).
|
|
259
|
+
- `"Column"`: The column number (1-indexed) where the profanity starts.
|
|
260
|
+
- `"Word"`: The detected profanity word.
|
|
261
|
+
- `"Language"`: Indicates the source of the word list:
|
|
262
|
+
- `<LanguageName>` (e.g., "English"): If only a built-in language list was used.
|
|
263
|
+
- `"Custom"`: If `language=None` and only a `custom_words_list` was used.
|
|
264
|
+
- `"Custom + <LanguageName>"` (e.g., "Custom + English"): If both a built-in list and `custom_words_list` were used.
|
|
265
|
+
- `"Custom + All"`: If `language='All'` and `custom_words_list` were used.
|
|
266
|
+
|
|
267
|
+
**4. Removing Profanity**
|
|
268
|
+
|
|
269
|
+
`remove_profanity` works similarly, accepting `language` and `custom_words_list` parameters.
|
|
270
|
+
|
|
271
|
+
```python
|
|
272
|
+
from valx import remove_profanity, load_custom_profanity_from_file
|
|
273
|
+
|
|
274
|
+
sample_text = ["This is fuck, custombadword1, and text with customfileword1."]
|
|
275
|
+
my_custom_words = ["custombadword1"]
|
|
276
|
+
custom_words_from_file = load_custom_profanity_from_file("my_profanity_file.txt") # Assuming it contains 'customfileword1'
|
|
277
|
+
|
|
278
|
+
# Remove profanity using English built-in + my_custom_words + custom_words_from_file
|
|
279
|
+
all_custom_words = list(set(my_custom_words + custom_words_from_file)) # Combine and unique
|
|
280
|
+
|
|
281
|
+
cleaned_text = remove_profanity(
|
|
282
|
+
sample_text,
|
|
283
|
+
output_file="cleaned_output.txt", # Optional: saves to file
|
|
284
|
+
language="English",
|
|
285
|
+
custom_words_list=all_custom_words
|
|
286
|
+
)
|
|
287
|
+
# cleaned_text will have "fuck", "custombadword1", and "customfileword1" replaced with "bad word".
|
|
288
|
+
# e.g., ["This is bad word, bad word, and text with bad word."]
|
|
289
|
+
print(cleaned_text)
|
|
290
|
+
```
|
|
291
|
+
|
|
292
|
+
The `load_profanity_words` function (used internally) also accepts `language` and `custom_words_list` if you need direct access to the word lists.
|
|
293
|
+
|
|
294
|
+
### Detect Sensitive Information
|
|
295
|
+
|
|
296
|
+
```python
|
|
297
|
+
from valx import detect_sensitive_information
|
|
298
|
+
|
|
299
|
+
# Detect sensitive information
|
|
300
|
+
detected_sensitive_info = detect_sensitive_information(sample_text)
|
|
301
|
+
```
|
|
302
|
+
|
|
303
|
+
> [!NOTE]
|
|
304
|
+
> We have updated this function, and it now includes an optional argument for `info_type`, which can be used to detect only specific types of sensitive information. It was also added to `remove_sensitive_information`.
|
|
305
|
+
|
|
306
|
+
### Remove Sensitive Information
|
|
307
|
+
|
|
308
|
+
```python
|
|
309
|
+
from valx import remove_sensitive_information
|
|
310
|
+
|
|
311
|
+
# Remove sensitive information
|
|
312
|
+
cleaned_text = remove_sensitive_information(sample_text2)
|
|
313
|
+
```
|
|
314
|
+
|
|
315
|
+
### Detect Hate Speech And Offensive Language
|
|
316
|
+
|
|
317
|
+
```python
|
|
318
|
+
from valx import detect_hate_speech
|
|
319
|
+
|
|
320
|
+
# Detect hate speech or offensive language
|
|
321
|
+
outcome_of_detection = detect_hate_speech("You are stupid.")
|
|
322
|
+
```
|
|
323
|
+
|
|
324
|
+
> [!IMPORTANT]
|
|
325
|
+
> The model's possible outputs are:
|
|
326
|
+
>
|
|
327
|
+
> - `['Hate Speech']`: The text was flagged and contained hate speech.
|
|
328
|
+
> - `['Offensive Speech']`: The text was flagged and contained offensive speech.
|
|
329
|
+
> - `['No Hate and Offensive Speech']`: The text was not flagged for any hate speech or offensive speech.
|
|
330
|
+
|
|
331
|
+
> [!NOTE]
|
|
332
|
+
> See our [official documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more examples on how to use **ValX**.
|
|
333
|
+
|
|
334
|
+
## Contributing
|
|
335
|
+
|
|
336
|
+
Contributions are welcome! If you encounter any issues, have suggestions, or want to contribute to ValX, please open an issue or submit a pull request on [GitHub](https://github.com/infinitode/valx).
|
|
337
|
+
|
|
338
|
+
## License
|
|
339
|
+
|
|
340
|
+
ValX is released under the terms of the **MIT License (Modified)**. Please see the [LICENSE](https://github.com/infinitode/valx/blob/main/LICENSE) file for the full text.
|
|
341
|
+
|
|
342
|
+
### Derived licenses
|
|
343
|
+
|
|
344
|
+
---
|
|
345
|
+
|
|
346
|
+
ValX uses data from this GitHub repository:
|
|
347
|
+
https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words/
|
|
348
|
+
© 2012-2020 Shutterstock, Inc.
|
|
349
|
+
|
|
350
|
+
Creative Commons Attribution 4.0 International License:
|
|
351
|
+
https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words/blob/master/LICENSE
|
|
352
|
+
|
|
353
|
+
---
|
|
354
|
+
|
|
355
|
+
**Modified License Clause**
|
|
356
|
+
|
|
357
|
+
The modified license clause grants users the permission to make derivative works based on the ValX software. However, it requires any substantial changes to the software to be clearly distinguished from the original work and distributed under a different name.
|
|
@@ -5,7 +5,7 @@ with open("README.md", "r", encoding="utf-8") as fh:
|
|
|
5
5
|
|
|
6
6
|
setup(
|
|
7
7
|
name='valx',
|
|
8
|
-
version='0.2.
|
|
8
|
+
version='0.2.6',
|
|
9
9
|
author='Infinitode Pty Ltd',
|
|
10
10
|
author_email='infinitode.ltd@gmail.com',
|
|
11
11
|
description='An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.',
|