valx 0.2.3__tar.gz → 0.2.5__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {valx-0.2.3/valx.egg-info → valx-0.2.5}/PKG-INFO +151 -11
- valx-0.2.5/README.md +314 -0
- {valx-0.2.3 → valx-0.2.5}/setup.py +3 -3
- {valx-0.2.3 → valx-0.2.5}/valx/__init__.py +109 -23
- valx-0.2.5/valx/models/classifier_upgraded.pkl +0 -0
- {valx-0.2.3 → valx-0.2.5/valx.egg-info}/PKG-INFO +151 -11
- {valx-0.2.3 → valx-0.2.5}/valx.egg-info/SOURCES.txt +1 -0
- valx-0.2.5/valx.egg-info/requires.txt +1 -0
- valx-0.2.3/README.md +0 -184
- valx-0.2.3/valx.egg-info/requires.txt +0 -1
- {valx-0.2.3 → valx-0.2.5}/LICENSE +0 -0
- {valx-0.2.3 → valx-0.2.5}/setup.cfg +0 -0
- {valx-0.2.3 → valx-0.2.5}/valx/models/count_vectorizer.sav +0 -0
- {valx-0.2.3 → valx-0.2.5}/valx/models/decision_tree_model.sav +0 -0
- {valx-0.2.3 → valx-0.2.5}/valx.egg-info/dependency_links.txt +0 -0
- {valx-0.2.3 → valx-0.2.5}/valx.egg-info/top_level.txt +0 -0
|
@@ -1,11 +1,11 @@
|
|
|
1
|
-
Metadata-Version: 2.
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
2
|
Name: valx
|
|
3
|
-
Version: 0.2.
|
|
3
|
+
Version: 0.2.5
|
|
4
4
|
Summary: An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.
|
|
5
5
|
Home-page: https://github.com/infinitode/valx
|
|
6
6
|
Author: Infinitode Pty Ltd
|
|
7
7
|
Author-email: infinitode.ltd@gmail.com
|
|
8
|
-
Classifier: Development Status ::
|
|
8
|
+
Classifier: Development Status :: 5 - Production/Stable
|
|
9
9
|
Classifier: Intended Audience :: Developers
|
|
10
10
|
Classifier: License :: OSI Approved :: MIT License
|
|
11
11
|
Classifier: Programming Language :: Python :: 3.6
|
|
@@ -18,7 +18,17 @@ Classifier: Programming Language :: Python :: 3.12
|
|
|
18
18
|
Requires-Python: >=3.6
|
|
19
19
|
Description-Content-Type: text/markdown
|
|
20
20
|
License-File: LICENSE
|
|
21
|
-
Requires-Dist: scikit-learn
|
|
21
|
+
Requires-Dist: scikit-learn
|
|
22
|
+
Dynamic: author
|
|
23
|
+
Dynamic: author-email
|
|
24
|
+
Dynamic: classifier
|
|
25
|
+
Dynamic: description
|
|
26
|
+
Dynamic: description-content-type
|
|
27
|
+
Dynamic: home-page
|
|
28
|
+
Dynamic: license-file
|
|
29
|
+
Dynamic: requires-dist
|
|
30
|
+
Dynamic: requires-python
|
|
31
|
+
Dynamic: summary
|
|
22
32
|
|
|
23
33
|
# ValX
|
|
24
34
|

|
|
@@ -30,9 +40,35 @@ Requires-Dist: scikit-learn==1.2.2
|
|
|
30
40
|
An open-source Python library for data cleaning tasks. It includes functions for profanity detection, and removal, and detection and removal of personal information. Also includes hate speech and offensive language detection and removal, using AI.
|
|
31
41
|
|
|
32
42
|
> [!IMPORTANT]
|
|
43
|
+
> If you are using `scikit-learn` versions older than version `1.3.0`, please also downgrade your version of `numpy` as stated below. Otherwise, you can continue to use your preferred version of `scikit-learn` without downgrading `numpy`.
|
|
44
|
+
>
|
|
33
45
|
> Please downgrade to `numpy` version `1.26.4`. Our ValX **DecisionTreeClassifier** AI model, relies on lower versions of `numpy`, because it was trained on these versions.
|
|
34
46
|
> For more information see: https://techoverflow.net/2024/07/23/how-to-fix-numpy-dtype-size-changed-may-indicate-binary-incompatibility-expected-96-from-c-header-got-88-from-pyobject/
|
|
35
47
|
|
|
48
|
+
> [!NOTE]
|
|
49
|
+
> ValX will automatically install a version of `scikit-learn` that is compatible with your device if you don't have one already.
|
|
50
|
+
|
|
51
|
+
## Changes in 0.2.5
|
|
52
|
+
|
|
53
|
+
ValX v0.2.5 introduces enhanced flexibility for profanity filtering by adding support for custom profanity lists:
|
|
54
|
+
|
|
55
|
+
- **Custom Profanity Word Lists**: Users can now provide their own lists of profane words directly as Python lists to the `detect_profanity` and `remove_profanity` functions via the new `custom_words_list` parameter.
|
|
56
|
+
- **Standalone Custom Lists**: Utilize your custom profanity list exclusively by setting the `language` parameter to `None`. ValX will then only use the words provided in `custom_words_list`.
|
|
57
|
+
- **Combined Lists**: Use a custom list in conjunction with ValX's built-in language-specific wordlists. Simply provide both a `language` (e.g., "English") and your `custom_words_list`. ValX will use the combined set of words.
|
|
58
|
+
- **Loading Custom Lists from File**: A new helper function, `load_custom_profanity_from_file(filepath)`, allows you to easily load custom profanity words from a text file.
|
|
59
|
+
- **File Format**: The file should contain one profanity word per line.
|
|
60
|
+
- Lines starting with a hash symbol (`#`) are treated as comments and ignored.
|
|
61
|
+
- Empty lines or lines containing only whitespace are also ignored.
|
|
62
|
+
- **Updated Detection Reporting**: The `detect_profanity` function's output now specifies the source of detected profanity more clearly (e.g., "Custom", "Custom + English").
|
|
63
|
+
|
|
64
|
+
These features give users greater control over the profanity filtering process, allowing for more tailored and specific use cases.
|
|
65
|
+
|
|
66
|
+
## Changes in 0.2.4
|
|
67
|
+
|
|
68
|
+
Fixed a major incompatibility issue with `scikit-learn` due to version changes in `scikit-learn v1.3.0` which causes compatibility issues with versions later than `1.2.2`. ValX can now be used with `scikit-learn` versions earlier and later than `1.3.0`!
|
|
69
|
+
|
|
70
|
+
We've also removed `scikit-learn==1.2.2` as a dependency, as most versions of `scikit-learn` will now work.
|
|
71
|
+
|
|
36
72
|
## Changes in 0.2.3
|
|
37
73
|
|
|
38
74
|
We have introduced a new optional `info_type` parameter into our `detect_sensitive_information`, and `remove_sensitive_information` functions, to allow you to have fine-grained control over what sensitive information you want to detect or remove.
|
|
@@ -124,25 +160,129 @@ Below is a complete list of all the available supported languages for ValX's pro
|
|
|
124
160
|
|
|
125
161
|
## Usage
|
|
126
162
|
|
|
127
|
-
###
|
|
163
|
+
### Profanity Detection and Removal
|
|
164
|
+
|
|
165
|
+
ValX allows for flexible profanity filtering using built-in language lists, custom word lists (provided as Python lists or loaded from files), or a combination of both.
|
|
166
|
+
|
|
167
|
+
**1. Basic Profanity Detection (Built-in Language)**
|
|
128
168
|
|
|
129
169
|
```python
|
|
130
170
|
from valx import detect_profanity
|
|
131
171
|
|
|
132
|
-
|
|
172
|
+
sample_text = ["This is some fuck and porn text."]
|
|
173
|
+
# Detect profanity using the English list
|
|
133
174
|
results = detect_profanity(sample_text, language='English')
|
|
134
|
-
|
|
175
|
+
# results will be:
|
|
176
|
+
# [
|
|
177
|
+
# {'Line': 1, 'Column': 14, 'Word': 'fuck', 'Language': 'English'},
|
|
178
|
+
# {'Line': 1, 'Column': 23, 'Word': 'porn', 'Language': 'English'}
|
|
179
|
+
# ]
|
|
180
|
+
print(results)
|
|
135
181
|
```
|
|
136
182
|
|
|
137
|
-
|
|
183
|
+
**2. Profanity Detection with a Custom Word List (Python List)**
|
|
184
|
+
|
|
185
|
+
You can provide your own list of words to filter.
|
|
138
186
|
|
|
139
187
|
```python
|
|
140
|
-
from valx import
|
|
188
|
+
from valx import detect_profanity
|
|
141
189
|
|
|
142
|
-
|
|
143
|
-
|
|
190
|
+
sample_text = ["This contains custombadword1 and also asshole from English list."]
|
|
191
|
+
my_custom_words = ["custombadword1", "anothercustom"]
|
|
192
|
+
|
|
193
|
+
# Option A: Custom list ONLY (language=None)
|
|
194
|
+
results_custom_only = detect_profanity(sample_text, language=None, custom_words_list=my_custom_words)
|
|
195
|
+
# results_custom_only will detect "custombadword1" with Language: "Custom"
|
|
196
|
+
# [{'Line': 1, 'Column': 15, 'Word': 'custombadword1', 'Language': 'Custom'}]
|
|
197
|
+
print(results_custom_only)
|
|
198
|
+
|
|
199
|
+
# Option B: Custom list COMBINED with a built-in language
|
|
200
|
+
results_custom_plus_english = detect_profanity(sample_text, language="English", custom_words_list=my_custom_words)
|
|
201
|
+
# results_custom_plus_english will detect "custombadword1" and "asshole"
|
|
202
|
+
# Language will be "Custom + English"
|
|
203
|
+
# [
|
|
204
|
+
# {'Line': 1, 'Column': 15, 'Word': 'custombadword1', 'Language': 'Custom + English'},
|
|
205
|
+
# {'Line': 1, 'Column': 43, 'Word': 'asshole', 'Language': 'Custom + English'}
|
|
206
|
+
# ]
|
|
207
|
+
print(results_custom_plus_english)
|
|
144
208
|
```
|
|
145
209
|
|
|
210
|
+
**3. Loading Custom Profanity Words from a File**
|
|
211
|
+
|
|
212
|
+
ValX provides a helper function to load words from a text file (one word per line, '#' for comments).
|
|
213
|
+
|
|
214
|
+
```python
|
|
215
|
+
from valx import detect_profanity, load_custom_profanity_from_file
|
|
216
|
+
|
|
217
|
+
# Assume 'my_profanity_file.txt' contains:
|
|
218
|
+
# customfileword1
|
|
219
|
+
# # this is a comment
|
|
220
|
+
# customfileword2
|
|
221
|
+
|
|
222
|
+
custom_words_from_file = load_custom_profanity_from_file("my_profanity_file.txt")
|
|
223
|
+
# custom_words_from_file will be: ['customfileword1', 'customfileword2']
|
|
224
|
+
|
|
225
|
+
sample_text_for_file = ["Text with customfileword1 and built-in shit."]
|
|
226
|
+
|
|
227
|
+
# Use file-loaded list with English built-in list
|
|
228
|
+
results_file_plus_english = detect_profanity(
|
|
229
|
+
sample_text_for_file,
|
|
230
|
+
language="English",
|
|
231
|
+
custom_words_list=custom_words_from_file
|
|
232
|
+
)
|
|
233
|
+
# Detects "customfileword1" and "shit", Language: "Custom + English"
|
|
234
|
+
print(results_file_plus_english)
|
|
235
|
+
|
|
236
|
+
# Use file-loaded list ONLY
|
|
237
|
+
results_file_only = detect_profanity(
|
|
238
|
+
sample_text_for_file,
|
|
239
|
+
language=None, # Important: set language to None
|
|
240
|
+
custom_words_list=custom_words_from_file
|
|
241
|
+
)
|
|
242
|
+
# Detects only "customfileword1", Language: "Custom"
|
|
243
|
+
print(results_file_only)
|
|
244
|
+
```
|
|
245
|
+
|
|
246
|
+
**Output Format for `detect_profanity`**
|
|
247
|
+
|
|
248
|
+
The `detect_profanity` function returns a list of dictionaries. Each dictionary includes:
|
|
249
|
+
- `"Line"`: The line number (1-indexed).
|
|
250
|
+
- `"Column"`: The column number (1-indexed) where the profanity starts.
|
|
251
|
+
- `"Word"`: The detected profanity word.
|
|
252
|
+
- `"Language"`: Indicates the source of the word list:
|
|
253
|
+
- `<LanguageName>` (e.g., "English"): If only a built-in language list was used.
|
|
254
|
+
- `"Custom"`: If `language=None` and only a `custom_words_list` was used.
|
|
255
|
+
- `"Custom + <LanguageName>"` (e.g., "Custom + English"): If both a built-in list and `custom_words_list` were used.
|
|
256
|
+
- `"Custom + All"`: If `language='All'` and `custom_words_list` were used.
|
|
257
|
+
|
|
258
|
+
|
|
259
|
+
**4. Removing Profanity**
|
|
260
|
+
|
|
261
|
+
`remove_profanity` works similarly, accepting `language` and `custom_words_list` parameters.
|
|
262
|
+
|
|
263
|
+
```python
|
|
264
|
+
from valx import remove_profanity, load_custom_profanity_from_file
|
|
265
|
+
|
|
266
|
+
sample_text = ["This is fuck, custombadword1, and text with customfileword1."]
|
|
267
|
+
my_custom_words = ["custombadword1"]
|
|
268
|
+
custom_words_from_file = load_custom_profanity_from_file("my_profanity_file.txt") # Assuming it contains 'customfileword1'
|
|
269
|
+
|
|
270
|
+
# Remove profanity using English built-in + my_custom_words + custom_words_from_file
|
|
271
|
+
all_custom_words = list(set(my_custom_words + custom_words_from_file)) # Combine and unique
|
|
272
|
+
|
|
273
|
+
cleaned_text = remove_profanity(
|
|
274
|
+
sample_text,
|
|
275
|
+
output_file="cleaned_output.txt", # Optional: saves to file
|
|
276
|
+
language="English",
|
|
277
|
+
custom_words_list=all_custom_words
|
|
278
|
+
)
|
|
279
|
+
# cleaned_text will have "fuck", "custombadword1", and "customfileword1" replaced with "bad word".
|
|
280
|
+
# e.g., ["This is bad word, bad word, and text with bad word."]
|
|
281
|
+
print(cleaned_text)
|
|
282
|
+
```
|
|
283
|
+
|
|
284
|
+
The `load_profanity_words` function (used internally) also accepts `language` and `custom_words_list` if you need direct access to the word lists.
|
|
285
|
+
|
|
146
286
|
### Detect Sensitive Information
|
|
147
287
|
|
|
148
288
|
```python
|
valx-0.2.5/README.md
ADDED
|
@@ -0,0 +1,314 @@
|
|
|
1
|
+
# ValX
|
|
2
|
+

|
|
3
|
+
[](https://github.com/infinitode/valx)
|
|
4
|
+

|
|
5
|
+

|
|
6
|
+

|
|
7
|
+
|
|
8
|
+
An open-source Python library for data cleaning tasks. It includes functions for profanity detection, and removal, and detection and removal of personal information. Also includes hate speech and offensive language detection and removal, using AI.
|
|
9
|
+
|
|
10
|
+
> [!IMPORTANT]
|
|
11
|
+
> If you are using `scikit-learn` versions older than version `1.3.0`, please also downgrade your version of `numpy` as stated below. Otherwise, you can continue to use your preferred version of `scikit-learn` without downgrading `numpy`.
|
|
12
|
+
>
|
|
13
|
+
> Please downgrade to `numpy` version `1.26.4`. Our ValX **DecisionTreeClassifier** AI model, relies on lower versions of `numpy`, because it was trained on these versions.
|
|
14
|
+
> For more information see: https://techoverflow.net/2024/07/23/how-to-fix-numpy-dtype-size-changed-may-indicate-binary-incompatibility-expected-96-from-c-header-got-88-from-pyobject/
|
|
15
|
+
|
|
16
|
+
> [!NOTE]
|
|
17
|
+
> ValX will automatically install a version of `scikit-learn` that is compatible with your device if you don't have one already.
|
|
18
|
+
|
|
19
|
+
## Changes in 0.2.5
|
|
20
|
+
|
|
21
|
+
ValX v0.2.5 introduces enhanced flexibility for profanity filtering by adding support for custom profanity lists:
|
|
22
|
+
|
|
23
|
+
- **Custom Profanity Word Lists**: Users can now provide their own lists of profane words directly as Python lists to the `detect_profanity` and `remove_profanity` functions via the new `custom_words_list` parameter.
|
|
24
|
+
- **Standalone Custom Lists**: Utilize your custom profanity list exclusively by setting the `language` parameter to `None`. ValX will then only use the words provided in `custom_words_list`.
|
|
25
|
+
- **Combined Lists**: Use a custom list in conjunction with ValX's built-in language-specific wordlists. Simply provide both a `language` (e.g., "English") and your `custom_words_list`. ValX will use the combined set of words.
|
|
26
|
+
- **Loading Custom Lists from File**: A new helper function, `load_custom_profanity_from_file(filepath)`, allows you to easily load custom profanity words from a text file.
|
|
27
|
+
- **File Format**: The file should contain one profanity word per line.
|
|
28
|
+
- Lines starting with a hash symbol (`#`) are treated as comments and ignored.
|
|
29
|
+
- Empty lines or lines containing only whitespace are also ignored.
|
|
30
|
+
- **Updated Detection Reporting**: The `detect_profanity` function's output now specifies the source of detected profanity more clearly (e.g., "Custom", "Custom + English").
|
|
31
|
+
|
|
32
|
+
These features give users greater control over the profanity filtering process, allowing for more tailored and specific use cases.
|
|
33
|
+
|
|
34
|
+
## Changes in 0.2.4
|
|
35
|
+
|
|
36
|
+
Fixed a major incompatibility issue with `scikit-learn` due to version changes in `scikit-learn v1.3.0` which causes compatibility issues with versions later than `1.2.2`. ValX can now be used with `scikit-learn` versions earlier and later than `1.3.0`!
|
|
37
|
+
|
|
38
|
+
We've also removed `scikit-learn==1.2.2` as a dependency, as most versions of `scikit-learn` will now work.
|
|
39
|
+
|
|
40
|
+
## Changes in 0.2.3
|
|
41
|
+
|
|
42
|
+
We have introduced a new optional `info_type` parameter into our `detect_sensitive_information`, and `remove_sensitive_information` functions, to allow you to have fine-grained control over what sensitive information you want to detect or remove.
|
|
43
|
+
|
|
44
|
+
Also introduced more detection patterns for other types of sensitive information, including:
|
|
45
|
+
- `"iban"`: International Bank Account Number.
|
|
46
|
+
- `"mrn"`: Medical Record Number (may not work correctly, depending on provider and country).
|
|
47
|
+
- `"icd10"`: International Classification of Diseases, Tenth Revision.
|
|
48
|
+
- `"geo_coords"`: Geo-coordinates (latitude and longitude in decimal degrees format).
|
|
49
|
+
- `"username"`: Username handles (@username).
|
|
50
|
+
- `"file_path"`: File paths (general patterns for both Windows and Unix paths).
|
|
51
|
+
- `"bitcoin_wallet"`: Cryptocurrency wallet address.
|
|
52
|
+
- `"ethereum_wallet"`: Cryptocurrency wallet addresses.
|
|
53
|
+
|
|
54
|
+
## Changes in 0.2.2
|
|
55
|
+
|
|
56
|
+
We have refactored and changed the `detect_profanity` function:
|
|
57
|
+
- Removed unnecessary printing
|
|
58
|
+
- Now returns more information about each found profanity, including `Line`, `Column`, `Word`, and `Language`.
|
|
59
|
+
|
|
60
|
+
> [!NOTE]
|
|
61
|
+
> You can view [ValX's package documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more information on changes.
|
|
62
|
+
|
|
63
|
+
## Changes in 0.2.1
|
|
64
|
+
|
|
65
|
+
Using the AI models in ValX, you can now automatically remove hate speech, or offensive speech from your text data, without needing to run detection and write your own custom implementation method.
|
|
66
|
+
|
|
67
|
+
## Installation
|
|
68
|
+
|
|
69
|
+
You can install ValX using pip:
|
|
70
|
+
|
|
71
|
+
```bash
|
|
72
|
+
pip install valx
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
## Supported Python Versions
|
|
76
|
+
|
|
77
|
+
ValX supports the following Python versions:
|
|
78
|
+
|
|
79
|
+
- Python 3.6
|
|
80
|
+
- Python 3.7
|
|
81
|
+
- Python 3.8
|
|
82
|
+
- Python 3.9
|
|
83
|
+
- Python 3.10
|
|
84
|
+
- Python 3.11/Later (Preferred)
|
|
85
|
+
|
|
86
|
+
Please ensure that you have one of these Python versions installed before using ValX. ValX may not work as expected on lower versions of Python than the supported.
|
|
87
|
+
|
|
88
|
+
## Features
|
|
89
|
+
|
|
90
|
+
- **Profanity Detection**: Detect profane and NSFW words or terms.
|
|
91
|
+
- **Remove Profanity**: Remove profane and NSFW words or terms.
|
|
92
|
+
- **Detect Sensitive Information**: Detect sensitive information in text data.
|
|
93
|
+
- **Remove Sensitive Information**: Remove sensitive information from text data.
|
|
94
|
+
- **Detect Hate Speech**: Detect hate speech or offensive speech in text, using AI.
|
|
95
|
+
- **Remove Hate Speech**: Remove hate speech or offensive speech in text, using AI.
|
|
96
|
+
|
|
97
|
+
### List of supported languages for profanity detection and removal
|
|
98
|
+
Below is a complete list of all the available supported languages for ValX's profanity detection and removal functions which are valid values for `language`:
|
|
99
|
+
|
|
100
|
+
- **All**
|
|
101
|
+
- Arabic
|
|
102
|
+
- Czech
|
|
103
|
+
- Danish
|
|
104
|
+
- German
|
|
105
|
+
- English
|
|
106
|
+
- Esperanto
|
|
107
|
+
- Persian
|
|
108
|
+
- Finnish
|
|
109
|
+
- Filipino
|
|
110
|
+
- French
|
|
111
|
+
- French (CA)
|
|
112
|
+
- Hindi
|
|
113
|
+
- Hungarian
|
|
114
|
+
- Italian
|
|
115
|
+
- Japanese
|
|
116
|
+
- Kabyle
|
|
117
|
+
- Korean
|
|
118
|
+
- Dutch
|
|
119
|
+
- Norwegian
|
|
120
|
+
- Polish
|
|
121
|
+
- Portuguese
|
|
122
|
+
- Russian
|
|
123
|
+
- Swedish
|
|
124
|
+
- Thai
|
|
125
|
+
- Klingon
|
|
126
|
+
- Turkish
|
|
127
|
+
- Chinese
|
|
128
|
+
|
|
129
|
+
## Usage
|
|
130
|
+
|
|
131
|
+
### Profanity Detection and Removal
|
|
132
|
+
|
|
133
|
+
ValX allows for flexible profanity filtering using built-in language lists, custom word lists (provided as Python lists or loaded from files), or a combination of both.
|
|
134
|
+
|
|
135
|
+
**1. Basic Profanity Detection (Built-in Language)**
|
|
136
|
+
|
|
137
|
+
```python
|
|
138
|
+
from valx import detect_profanity
|
|
139
|
+
|
|
140
|
+
sample_text = ["This is some fuck and porn text."]
|
|
141
|
+
# Detect profanity using the English list
|
|
142
|
+
results = detect_profanity(sample_text, language='English')
|
|
143
|
+
# results will be:
|
|
144
|
+
# [
|
|
145
|
+
# {'Line': 1, 'Column': 14, 'Word': 'fuck', 'Language': 'English'},
|
|
146
|
+
# {'Line': 1, 'Column': 23, 'Word': 'porn', 'Language': 'English'}
|
|
147
|
+
# ]
|
|
148
|
+
print(results)
|
|
149
|
+
```
|
|
150
|
+
|
|
151
|
+
**2. Profanity Detection with a Custom Word List (Python List)**
|
|
152
|
+
|
|
153
|
+
You can provide your own list of words to filter.
|
|
154
|
+
|
|
155
|
+
```python
|
|
156
|
+
from valx import detect_profanity
|
|
157
|
+
|
|
158
|
+
sample_text = ["This contains custombadword1 and also asshole from English list."]
|
|
159
|
+
my_custom_words = ["custombadword1", "anothercustom"]
|
|
160
|
+
|
|
161
|
+
# Option A: Custom list ONLY (language=None)
|
|
162
|
+
results_custom_only = detect_profanity(sample_text, language=None, custom_words_list=my_custom_words)
|
|
163
|
+
# results_custom_only will detect "custombadword1" with Language: "Custom"
|
|
164
|
+
# [{'Line': 1, 'Column': 15, 'Word': 'custombadword1', 'Language': 'Custom'}]
|
|
165
|
+
print(results_custom_only)
|
|
166
|
+
|
|
167
|
+
# Option B: Custom list COMBINED with a built-in language
|
|
168
|
+
results_custom_plus_english = detect_profanity(sample_text, language="English", custom_words_list=my_custom_words)
|
|
169
|
+
# results_custom_plus_english will detect "custombadword1" and "asshole"
|
|
170
|
+
# Language will be "Custom + English"
|
|
171
|
+
# [
|
|
172
|
+
# {'Line': 1, 'Column': 15, 'Word': 'custombadword1', 'Language': 'Custom + English'},
|
|
173
|
+
# {'Line': 1, 'Column': 43, 'Word': 'asshole', 'Language': 'Custom + English'}
|
|
174
|
+
# ]
|
|
175
|
+
print(results_custom_plus_english)
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
**3. Loading Custom Profanity Words from a File**
|
|
179
|
+
|
|
180
|
+
ValX provides a helper function to load words from a text file (one word per line, '#' for comments).
|
|
181
|
+
|
|
182
|
+
```python
|
|
183
|
+
from valx import detect_profanity, load_custom_profanity_from_file
|
|
184
|
+
|
|
185
|
+
# Assume 'my_profanity_file.txt' contains:
|
|
186
|
+
# customfileword1
|
|
187
|
+
# # this is a comment
|
|
188
|
+
# customfileword2
|
|
189
|
+
|
|
190
|
+
custom_words_from_file = load_custom_profanity_from_file("my_profanity_file.txt")
|
|
191
|
+
# custom_words_from_file will be: ['customfileword1', 'customfileword2']
|
|
192
|
+
|
|
193
|
+
sample_text_for_file = ["Text with customfileword1 and built-in shit."]
|
|
194
|
+
|
|
195
|
+
# Use file-loaded list with English built-in list
|
|
196
|
+
results_file_plus_english = detect_profanity(
|
|
197
|
+
sample_text_for_file,
|
|
198
|
+
language="English",
|
|
199
|
+
custom_words_list=custom_words_from_file
|
|
200
|
+
)
|
|
201
|
+
# Detects "customfileword1" and "shit", Language: "Custom + English"
|
|
202
|
+
print(results_file_plus_english)
|
|
203
|
+
|
|
204
|
+
# Use file-loaded list ONLY
|
|
205
|
+
results_file_only = detect_profanity(
|
|
206
|
+
sample_text_for_file,
|
|
207
|
+
language=None, # Important: set language to None
|
|
208
|
+
custom_words_list=custom_words_from_file
|
|
209
|
+
)
|
|
210
|
+
# Detects only "customfileword1", Language: "Custom"
|
|
211
|
+
print(results_file_only)
|
|
212
|
+
```
|
|
213
|
+
|
|
214
|
+
**Output Format for `detect_profanity`**
|
|
215
|
+
|
|
216
|
+
The `detect_profanity` function returns a list of dictionaries. Each dictionary includes:
|
|
217
|
+
- `"Line"`: The line number (1-indexed).
|
|
218
|
+
- `"Column"`: The column number (1-indexed) where the profanity starts.
|
|
219
|
+
- `"Word"`: The detected profanity word.
|
|
220
|
+
- `"Language"`: Indicates the source of the word list:
|
|
221
|
+
- `<LanguageName>` (e.g., "English"): If only a built-in language list was used.
|
|
222
|
+
- `"Custom"`: If `language=None` and only a `custom_words_list` was used.
|
|
223
|
+
- `"Custom + <LanguageName>"` (e.g., "Custom + English"): If both a built-in list and `custom_words_list` were used.
|
|
224
|
+
- `"Custom + All"`: If `language='All'` and `custom_words_list` were used.
|
|
225
|
+
|
|
226
|
+
|
|
227
|
+
**4. Removing Profanity**
|
|
228
|
+
|
|
229
|
+
`remove_profanity` works similarly, accepting `language` and `custom_words_list` parameters.
|
|
230
|
+
|
|
231
|
+
```python
|
|
232
|
+
from valx import remove_profanity, load_custom_profanity_from_file
|
|
233
|
+
|
|
234
|
+
sample_text = ["This is fuck, custombadword1, and text with customfileword1."]
|
|
235
|
+
my_custom_words = ["custombadword1"]
|
|
236
|
+
custom_words_from_file = load_custom_profanity_from_file("my_profanity_file.txt") # Assuming it contains 'customfileword1'
|
|
237
|
+
|
|
238
|
+
# Remove profanity using English built-in + my_custom_words + custom_words_from_file
|
|
239
|
+
all_custom_words = list(set(my_custom_words + custom_words_from_file)) # Combine and unique
|
|
240
|
+
|
|
241
|
+
cleaned_text = remove_profanity(
|
|
242
|
+
sample_text,
|
|
243
|
+
output_file="cleaned_output.txt", # Optional: saves to file
|
|
244
|
+
language="English",
|
|
245
|
+
custom_words_list=all_custom_words
|
|
246
|
+
)
|
|
247
|
+
# cleaned_text will have "fuck", "custombadword1", and "customfileword1" replaced with "bad word".
|
|
248
|
+
# e.g., ["This is bad word, bad word, and text with bad word."]
|
|
249
|
+
print(cleaned_text)
|
|
250
|
+
```
|
|
251
|
+
|
|
252
|
+
The `load_profanity_words` function (used internally) also accepts `language` and `custom_words_list` if you need direct access to the word lists.
|
|
253
|
+
|
|
254
|
+
### Detect Sensitive Information
|
|
255
|
+
|
|
256
|
+
```python
|
|
257
|
+
from valx import detect_sensitive_information
|
|
258
|
+
|
|
259
|
+
# Detect sensitive information
|
|
260
|
+
detected_sensitive_info = detect_sensitive_information(sample_text)
|
|
261
|
+
```
|
|
262
|
+
|
|
263
|
+
> [!NOTE]
|
|
264
|
+
> We have updated this function, and it now includes an optional argument for `info_type`, which can be used to detect only specific types of sensitive information. It was also added to `remove_sensitive_information`.
|
|
265
|
+
|
|
266
|
+
### Remove Sensitive Information
|
|
267
|
+
|
|
268
|
+
```python
|
|
269
|
+
from valx import remove_sensitive_information
|
|
270
|
+
|
|
271
|
+
# Remove sensitive information
|
|
272
|
+
cleaned_text = remove_sensitive_information(sample_text2)
|
|
273
|
+
```
|
|
274
|
+
|
|
275
|
+
### Detect Hate Speech And Offensive Language
|
|
276
|
+
|
|
277
|
+
```python
|
|
278
|
+
from valx import detect_hate_speech
|
|
279
|
+
|
|
280
|
+
# Detect hate speech or offensive language
|
|
281
|
+
outcome_of_detection = detect_hate_speech("You are stupid.")
|
|
282
|
+
```
|
|
283
|
+
|
|
284
|
+
> [!IMPORTANT]
|
|
285
|
+
> The model's possible outputs are:
|
|
286
|
+
> - `['Hate Speech']`: The text was flagged and contained hate speech.
|
|
287
|
+
> - `['Offensive Speech']`: The text was flagged and contained offensive speech.
|
|
288
|
+
> - `['No Hate and Offensive Speech']`: The text was not flagged for any hate speech or offensive speech.
|
|
289
|
+
|
|
290
|
+
> [!NOTE]
|
|
291
|
+
> See our [official documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more examples on how to use **ValX**.
|
|
292
|
+
|
|
293
|
+
## Contributing
|
|
294
|
+
|
|
295
|
+
Contributions are welcome! If you encounter any issues, have suggestions, or want to contribute to ValX, please open an issue or submit a pull request on [GitHub](https://github.com/infinitode/valx).
|
|
296
|
+
|
|
297
|
+
## License
|
|
298
|
+
|
|
299
|
+
ValX is released under the terms of the **MIT License (Modified)**. Please see the [LICENSE](https://github.com/infinitode/valx/blob/main/LICENSE) file for the full text.
|
|
300
|
+
|
|
301
|
+
### Derived licenses
|
|
302
|
+
---
|
|
303
|
+
ValX uses data from this GitHub repository:
|
|
304
|
+
https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words/
|
|
305
|
+
© 2012-2020 Shutterstock, Inc.
|
|
306
|
+
|
|
307
|
+
Creative Commons Attribution 4.0 International License:
|
|
308
|
+
https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words/blob/master/LICENSE
|
|
309
|
+
|
|
310
|
+
---
|
|
311
|
+
|
|
312
|
+
**Modified License Clause**
|
|
313
|
+
|
|
314
|
+
The modified license clause grants users the permission to make derivative works based on the ValX software. However, it requires any substantial changes to the software to be clearly distinguished from the original work and distributed under a different name.
|
|
@@ -5,7 +5,7 @@ with open("README.md", "r", encoding="utf-8") as fh:
|
|
|
5
5
|
|
|
6
6
|
setup(
|
|
7
7
|
name='valx',
|
|
8
|
-
version='0.2.
|
|
8
|
+
version='0.2.5',
|
|
9
9
|
author='Infinitode Pty Ltd',
|
|
10
10
|
author_email='infinitode.ltd@gmail.com',
|
|
11
11
|
description='An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.',
|
|
@@ -15,10 +15,10 @@ setup(
|
|
|
15
15
|
packages=find_packages(),
|
|
16
16
|
package_data={'valx': ['models/*']},
|
|
17
17
|
install_requires=[
|
|
18
|
-
'scikit-learn
|
|
18
|
+
'scikit-learn'
|
|
19
19
|
],
|
|
20
20
|
classifiers=[
|
|
21
|
-
'Development Status ::
|
|
21
|
+
'Development Status :: 5 - Production/Stable',
|
|
22
22
|
'Intended Audience :: Developers',
|
|
23
23
|
'License :: OSI Approved :: MIT License',
|
|
24
24
|
'Programming Language :: Python :: 3.6',
|
|
@@ -3,6 +3,7 @@ import re
|
|
|
3
3
|
|
|
4
4
|
# New version, that includes AI detection
|
|
5
5
|
import pickle
|
|
6
|
+
from sklearn import __version__ as sklearnversion
|
|
6
7
|
from sklearn.tree import DecisionTreeClassifier # Import the DecisionTreeClassifier class
|
|
7
8
|
from sklearn.feature_extraction.text import CountVectorizer # Import the CountVectorizer class
|
|
8
9
|
|
|
@@ -15,15 +16,20 @@ Creative Commons Attribution 4.0 International License:
|
|
|
15
16
|
https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words/blob/master/LICENSE
|
|
16
17
|
'''
|
|
17
18
|
|
|
18
|
-
def load_profanity_words(language='English'):
|
|
19
|
+
def load_profanity_words(language='English', custom_words_list: list[str] = None):
|
|
19
20
|
"""
|
|
20
|
-
Load profanity words for the specified language.
|
|
21
|
+
Load profanity words for the specified language, optionally including a custom list.
|
|
21
22
|
|
|
22
23
|
Args:
|
|
23
|
-
language (str): The language for which to load profanity words.
|
|
24
|
+
language (str, optional): The language for which to load profanity words.
|
|
25
|
+
Defaults to 'English'. Can be None to use only custom_words_list.
|
|
26
|
+
custom_words_list (list[str], optional): A list of custom profanity words. Defaults to None.
|
|
24
27
|
|
|
25
28
|
Returns:
|
|
26
|
-
list: A list of profanity words
|
|
29
|
+
list: A list of profanity words.
|
|
30
|
+
|
|
31
|
+
Raises:
|
|
32
|
+
ValueError: If language is None and custom_words_list is None or empty.
|
|
27
33
|
"""
|
|
28
34
|
# Profanity words as a string
|
|
29
35
|
PROFANITY_WORDS = """
|
|
@@ -2739,35 +2745,102 @@ def load_profanity_words(language='English'):
|
|
|
2739
2745
|
if current_language:
|
|
2740
2746
|
profanity_lists[current_language] = current_words
|
|
2741
2747
|
|
|
2742
|
-
|
|
2743
|
-
|
|
2748
|
+
loaded_words = []
|
|
2749
|
+
if language is None:
|
|
2750
|
+
if not custom_words_list:
|
|
2751
|
+
raise ValueError("If language is None, custom_words_list must be provided and non-empty.")
|
|
2752
|
+
loaded_words = list(custom_words_list) # Use a copy
|
|
2753
|
+
elif language == 'All':
|
|
2744
2754
|
for words in profanity_lists.values():
|
|
2745
|
-
|
|
2746
|
-
|
|
2755
|
+
loaded_words.extend(words)
|
|
2756
|
+
if custom_words_list:
|
|
2757
|
+
loaded_words.extend(w for w in custom_words_list if w not in loaded_words)
|
|
2747
2758
|
else:
|
|
2748
|
-
|
|
2759
|
+
loaded_words = list(profanity_lists.get(language, [])) # Use a copy
|
|
2760
|
+
if custom_words_list:
|
|
2761
|
+
loaded_words.extend(w for w in custom_words_list if w not in loaded_words)
|
|
2762
|
+
|
|
2763
|
+
return list(set(loaded_words)) # Return unique words
|
|
2764
|
+
|
|
2765
|
+
|
|
2766
|
+
def load_custom_profanity_from_file(filepath: str) -> list[str]:
|
|
2767
|
+
"""
|
|
2768
|
+
Loads a custom list of profanity words from a file.
|
|
2749
2769
|
|
|
2770
|
+
The file should contain one word per line. Lines starting with '#' will be ignored as comments.
|
|
2771
|
+
Empty lines or lines with only whitespace will also be ignored.
|
|
2772
|
+
|
|
2773
|
+
Args:
|
|
2774
|
+
filepath (str): The path to the file containing custom profanity words.
|
|
2775
|
+
|
|
2776
|
+
Returns:
|
|
2777
|
+
list[str]: A list of words loaded from the file.
|
|
2750
2778
|
|
|
2751
|
-
|
|
2779
|
+
Raises:
|
|
2780
|
+
FileNotFoundError: If the specified filepath does not exist.
|
|
2781
|
+
IOError: For other file reading issues.
|
|
2782
|
+
"""
|
|
2783
|
+
custom_words = []
|
|
2784
|
+
try:
|
|
2785
|
+
with open(filepath, 'r', encoding='utf-8') as f:
|
|
2786
|
+
for line in f:
|
|
2787
|
+
word = line.strip()
|
|
2788
|
+
if word and not word.startswith('#'):
|
|
2789
|
+
custom_words.append(word)
|
|
2790
|
+
except FileNotFoundError:
|
|
2791
|
+
# Re-raise FileNotFoundError to be explicit, or handle as per specific project policy
|
|
2792
|
+
# For example, could log a warning and return empty list:
|
|
2793
|
+
# print(f"Warning: File not found at {filepath}. Returning empty custom word list.")
|
|
2794
|
+
# return []
|
|
2795
|
+
raise
|
|
2796
|
+
except IOError as e:
|
|
2797
|
+
# Handle other IO errors, perhaps log them or raise a more generic error
|
|
2798
|
+
# print(f"Warning: Error reading file {filepath}: {e}. Returning empty custom word list.")
|
|
2799
|
+
# return []
|
|
2800
|
+
raise IOError(f"Error reading custom profanity file {filepath}: {e}")
|
|
2801
|
+
return custom_words
|
|
2802
|
+
|
|
2803
|
+
|
|
2804
|
+
def detect_profanity(text_data, language='English', custom_words_list: list[str] = None):
|
|
2752
2805
|
"""
|
|
2753
2806
|
Detect profanity in text data using regex.
|
|
2754
2807
|
|
|
2755
2808
|
Args:
|
|
2756
2809
|
text_data (list): A list of strings representing the text data to analyze.
|
|
2757
|
-
language (str): The language used to detect profanity. Defaults to 'English'.
|
|
2810
|
+
language (str, optional): The language used to detect profanity. Defaults to 'English'.
|
|
2811
|
+
Can be None if custom_words_list is provided.
|
|
2812
|
+
Available languages include: All, Arabic, Czech, Danish, German,
|
|
2813
|
+
English, Esperanto, Persian, Finnish, Filipino, French,
|
|
2814
|
+
French (CA), Hindi, Hungarian, Italian, Japanese, Kabyle,
|
|
2815
|
+
Korean, Dutch, Norwegian, Polish, Portuguese, Russian,
|
|
2816
|
+
Swedish, Thai, Klingon, Turkish, Chinese.
|
|
2817
|
+
custom_words_list (list[str], optional): A list of custom profanity words. Defaults to None.
|
|
2758
2818
|
Returns:
|
|
2759
2819
|
list: A list of dictionaries where each dictionary represents a detected instance of profanity.
|
|
2760
2820
|
Each dictionary contains the following keys:
|
|
2761
2821
|
- "Line" (int): The line number where the profanity was detected.
|
|
2762
2822
|
- "Column" (int): The column number (position in the line) where the profanity starts.
|
|
2763
2823
|
- "Word" (str): The detected profanity word.
|
|
2764
|
-
- "Language" (str): The language
|
|
2824
|
+
- "Language" (str): The language string indicating the source of the profanity list.
|
|
2765
2825
|
"""
|
|
2766
|
-
profanity_keywords = load_profanity_words(language)
|
|
2826
|
+
profanity_keywords = load_profanity_words(language, custom_words_list)
|
|
2767
2827
|
detected = set() # Use a set to store unique occurrences
|
|
2768
2828
|
|
|
2829
|
+
display_language = "Custom"
|
|
2830
|
+
if language:
|
|
2831
|
+
if custom_words_list:
|
|
2832
|
+
display_language = f"Custom + {language}"
|
|
2833
|
+
else:
|
|
2834
|
+
display_language = language
|
|
2835
|
+
elif not custom_words_list:
|
|
2836
|
+
# This case should ideally be prevented by load_profanity_words raising an error,
|
|
2837
|
+
# but as a fallback, if somehow load_profanity_words returns empty and language is None
|
|
2838
|
+
# and no custom_words_list, we avoid error here.
|
|
2839
|
+
return []
|
|
2840
|
+
|
|
2841
|
+
|
|
2769
2842
|
for i, line in enumerate(text_data):
|
|
2770
|
-
# Skip lines with language markers
|
|
2843
|
+
# Skip lines with language markers (though less relevant if custom lists are primary)
|
|
2771
2844
|
if line.startswith("$") and line.endswith("$"):
|
|
2772
2845
|
continue
|
|
2773
2846
|
|
|
@@ -2775,36 +2848,39 @@ def detect_profanity(text_data, language='English'):
|
|
|
2775
2848
|
for profanity in profanity_keywords:
|
|
2776
2849
|
matches = re.finditer(r'\b{}\b'.format(re.escape(profanity)), line, flags=re.IGNORECASE)
|
|
2777
2850
|
for match in matches:
|
|
2778
|
-
|
|
2851
|
+
# Store with display_language, not the input `language` parameter
|
|
2852
|
+
detected.add((i + 1, match.start() + 1, display_language, profanity))
|
|
2779
2853
|
|
|
2780
2854
|
detections = [] # Initialize detections as a list
|
|
2781
2855
|
|
|
2782
2856
|
if detected:
|
|
2783
|
-
for line_num, col_num,
|
|
2857
|
+
for line_num, col_num, lang_display_name, word in detected:
|
|
2784
2858
|
detection_info = {
|
|
2785
2859
|
"Line": line_num,
|
|
2786
2860
|
"Column": col_num,
|
|
2787
2861
|
"Word": word,
|
|
2788
|
-
"Language":
|
|
2862
|
+
"Language": lang_display_name
|
|
2789
2863
|
}
|
|
2790
|
-
detections.append(detection_info)
|
|
2864
|
+
detections.append(detection_info)
|
|
2791
2865
|
|
|
2792
2866
|
return detections
|
|
2793
2867
|
|
|
2794
2868
|
|
|
2795
|
-
def remove_profanity(text_data, output_file=None, language='English'):
|
|
2869
|
+
def remove_profanity(text_data, output_file=None, language='English', custom_words_list: list[str] = None):
|
|
2796
2870
|
"""
|
|
2797
2871
|
Remove profanity from text data.
|
|
2798
2872
|
|
|
2799
2873
|
Args:
|
|
2800
2874
|
text_data (list): A list of strings representing the text data to clean.
|
|
2801
|
-
output_file (str): The file path to write the cleaned data. If None, cleaned data is not written to a file.
|
|
2802
|
-
language (str): The language for which to remove profanity. Defaults to 'English'.
|
|
2875
|
+
output_file (str, optional): The file path to write the cleaned data. If None, cleaned data is not written to a file.
|
|
2876
|
+
language (str, optional): The language for which to remove profanity. Defaults to 'English'.
|
|
2877
|
+
Can be None if custom_words_list is provided.
|
|
2878
|
+
custom_words_list (list[str], optional): A list of custom profanity words. Defaults to None.
|
|
2803
2879
|
|
|
2804
2880
|
Returns:
|
|
2805
2881
|
list: A list of strings representing the cleaned text data.
|
|
2806
2882
|
"""
|
|
2807
|
-
profanity_keywords = load_profanity_words(language)
|
|
2883
|
+
profanity_keywords = load_profanity_words(language, custom_words_list)
|
|
2808
2884
|
cleaned_data = []
|
|
2809
2885
|
|
|
2810
2886
|
for line in text_data:
|
|
@@ -2902,8 +2978,18 @@ def remove_sensitive_information(text_data, output_file=None, info_type=["email"
|
|
|
2902
2978
|
|
|
2903
2979
|
# New version
|
|
2904
2980
|
MODEL_DIR = os.path.join(os.path.dirname(__file__), 'models')
|
|
2905
|
-
DECISION_TREE_MODEL_PATH = os.path.join(MODEL_DIR, 'decision_tree_model.sav')
|
|
2906
2981
|
COUNT_VECTORIZER_PATH = os.path.join(MODEL_DIR, 'count_vectorizer.sav')
|
|
2982
|
+
DEFAULT_MODEL_FILENAME = "decision_tree_model.sav"
|
|
2983
|
+
UPGRADED_MODEL_FILENAME = "classifier_upgraded.pkl"
|
|
2984
|
+
|
|
2985
|
+
# Check scikit-learn version
|
|
2986
|
+
SKLEARN_VERSION = tuple(map(int, sklearnversion.split(".")))
|
|
2987
|
+
|
|
2988
|
+
# Set the model path based on the version
|
|
2989
|
+
if SKLEARN_VERSION >= (1, 3, 0):
|
|
2990
|
+
DECISION_TREE_MODEL_PATH = os.path.join(MODEL_DIR, UPGRADED_MODEL_FILENAME)
|
|
2991
|
+
else:
|
|
2992
|
+
DECISION_TREE_MODEL_PATH = os.path.join(MODEL_DIR, DEFAULT_MODEL_FILENAME)
|
|
2907
2993
|
|
|
2908
2994
|
# Load the saved models
|
|
2909
2995
|
model = pickle.load(open(DECISION_TREE_MODEL_PATH, 'rb'))
|
|
Binary file
|
|
@@ -1,11 +1,11 @@
|
|
|
1
|
-
Metadata-Version: 2.
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
2
|
Name: valx
|
|
3
|
-
Version: 0.2.
|
|
3
|
+
Version: 0.2.5
|
|
4
4
|
Summary: An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.
|
|
5
5
|
Home-page: https://github.com/infinitode/valx
|
|
6
6
|
Author: Infinitode Pty Ltd
|
|
7
7
|
Author-email: infinitode.ltd@gmail.com
|
|
8
|
-
Classifier: Development Status ::
|
|
8
|
+
Classifier: Development Status :: 5 - Production/Stable
|
|
9
9
|
Classifier: Intended Audience :: Developers
|
|
10
10
|
Classifier: License :: OSI Approved :: MIT License
|
|
11
11
|
Classifier: Programming Language :: Python :: 3.6
|
|
@@ -18,7 +18,17 @@ Classifier: Programming Language :: Python :: 3.12
|
|
|
18
18
|
Requires-Python: >=3.6
|
|
19
19
|
Description-Content-Type: text/markdown
|
|
20
20
|
License-File: LICENSE
|
|
21
|
-
Requires-Dist: scikit-learn
|
|
21
|
+
Requires-Dist: scikit-learn
|
|
22
|
+
Dynamic: author
|
|
23
|
+
Dynamic: author-email
|
|
24
|
+
Dynamic: classifier
|
|
25
|
+
Dynamic: description
|
|
26
|
+
Dynamic: description-content-type
|
|
27
|
+
Dynamic: home-page
|
|
28
|
+
Dynamic: license-file
|
|
29
|
+
Dynamic: requires-dist
|
|
30
|
+
Dynamic: requires-python
|
|
31
|
+
Dynamic: summary
|
|
22
32
|
|
|
23
33
|
# ValX
|
|
24
34
|

|
|
@@ -30,9 +40,35 @@ Requires-Dist: scikit-learn==1.2.2
|
|
|
30
40
|
An open-source Python library for data cleaning tasks. It includes functions for profanity detection, and removal, and detection and removal of personal information. Also includes hate speech and offensive language detection and removal, using AI.
|
|
31
41
|
|
|
32
42
|
> [!IMPORTANT]
|
|
43
|
+
> If you are using `scikit-learn` versions older than version `1.3.0`, please also downgrade your version of `numpy` as stated below. Otherwise, you can continue to use your preferred version of `scikit-learn` without downgrading `numpy`.
|
|
44
|
+
>
|
|
33
45
|
> Please downgrade to `numpy` version `1.26.4`. Our ValX **DecisionTreeClassifier** AI model, relies on lower versions of `numpy`, because it was trained on these versions.
|
|
34
46
|
> For more information see: https://techoverflow.net/2024/07/23/how-to-fix-numpy-dtype-size-changed-may-indicate-binary-incompatibility-expected-96-from-c-header-got-88-from-pyobject/
|
|
35
47
|
|
|
48
|
+
> [!NOTE]
|
|
49
|
+
> ValX will automatically install a version of `scikit-learn` that is compatible with your device if you don't have one already.
|
|
50
|
+
|
|
51
|
+
## Changes in 0.2.5
|
|
52
|
+
|
|
53
|
+
ValX v0.2.5 introduces enhanced flexibility for profanity filtering by adding support for custom profanity lists:
|
|
54
|
+
|
|
55
|
+
- **Custom Profanity Word Lists**: Users can now provide their own lists of profane words directly as Python lists to the `detect_profanity` and `remove_profanity` functions via the new `custom_words_list` parameter.
|
|
56
|
+
- **Standalone Custom Lists**: Utilize your custom profanity list exclusively by setting the `language` parameter to `None`. ValX will then only use the words provided in `custom_words_list`.
|
|
57
|
+
- **Combined Lists**: Use a custom list in conjunction with ValX's built-in language-specific wordlists. Simply provide both a `language` (e.g., "English") and your `custom_words_list`. ValX will use the combined set of words.
|
|
58
|
+
- **Loading Custom Lists from File**: A new helper function, `load_custom_profanity_from_file(filepath)`, allows you to easily load custom profanity words from a text file.
|
|
59
|
+
- **File Format**: The file should contain one profanity word per line.
|
|
60
|
+
- Lines starting with a hash symbol (`#`) are treated as comments and ignored.
|
|
61
|
+
- Empty lines or lines containing only whitespace are also ignored.
|
|
62
|
+
- **Updated Detection Reporting**: The `detect_profanity` function's output now specifies the source of detected profanity more clearly (e.g., "Custom", "Custom + English").
|
|
63
|
+
|
|
64
|
+
These features give users greater control over the profanity filtering process, allowing for more tailored and specific use cases.
|
|
65
|
+
|
|
66
|
+
## Changes in 0.2.4
|
|
67
|
+
|
|
68
|
+
Fixed a major incompatibility issue with `scikit-learn` due to version changes in `scikit-learn v1.3.0` which causes compatibility issues with versions later than `1.2.2`. ValX can now be used with `scikit-learn` versions earlier and later than `1.3.0`!
|
|
69
|
+
|
|
70
|
+
We've also removed `scikit-learn==1.2.2` as a dependency, as most versions of `scikit-learn` will now work.
|
|
71
|
+
|
|
36
72
|
## Changes in 0.2.3
|
|
37
73
|
|
|
38
74
|
We have introduced a new optional `info_type` parameter into our `detect_sensitive_information`, and `remove_sensitive_information` functions, to allow you to have fine-grained control over what sensitive information you want to detect or remove.
|
|
@@ -124,25 +160,129 @@ Below is a complete list of all the available supported languages for ValX's pro
|
|
|
124
160
|
|
|
125
161
|
## Usage
|
|
126
162
|
|
|
127
|
-
###
|
|
163
|
+
### Profanity Detection and Removal
|
|
164
|
+
|
|
165
|
+
ValX allows for flexible profanity filtering using built-in language lists, custom word lists (provided as Python lists or loaded from files), or a combination of both.
|
|
166
|
+
|
|
167
|
+
**1. Basic Profanity Detection (Built-in Language)**
|
|
128
168
|
|
|
129
169
|
```python
|
|
130
170
|
from valx import detect_profanity
|
|
131
171
|
|
|
132
|
-
|
|
172
|
+
sample_text = ["This is some fuck and porn text."]
|
|
173
|
+
# Detect profanity using the English list
|
|
133
174
|
results = detect_profanity(sample_text, language='English')
|
|
134
|
-
|
|
175
|
+
# results will be:
|
|
176
|
+
# [
|
|
177
|
+
# {'Line': 1, 'Column': 14, 'Word': 'fuck', 'Language': 'English'},
|
|
178
|
+
# {'Line': 1, 'Column': 23, 'Word': 'porn', 'Language': 'English'}
|
|
179
|
+
# ]
|
|
180
|
+
print(results)
|
|
135
181
|
```
|
|
136
182
|
|
|
137
|
-
|
|
183
|
+
**2. Profanity Detection with a Custom Word List (Python List)**
|
|
184
|
+
|
|
185
|
+
You can provide your own list of words to filter.
|
|
138
186
|
|
|
139
187
|
```python
|
|
140
|
-
from valx import
|
|
188
|
+
from valx import detect_profanity
|
|
141
189
|
|
|
142
|
-
|
|
143
|
-
|
|
190
|
+
sample_text = ["This contains custombadword1 and also asshole from English list."]
|
|
191
|
+
my_custom_words = ["custombadword1", "anothercustom"]
|
|
192
|
+
|
|
193
|
+
# Option A: Custom list ONLY (language=None)
|
|
194
|
+
results_custom_only = detect_profanity(sample_text, language=None, custom_words_list=my_custom_words)
|
|
195
|
+
# results_custom_only will detect "custombadword1" with Language: "Custom"
|
|
196
|
+
# [{'Line': 1, 'Column': 15, 'Word': 'custombadword1', 'Language': 'Custom'}]
|
|
197
|
+
print(results_custom_only)
|
|
198
|
+
|
|
199
|
+
# Option B: Custom list COMBINED with a built-in language
|
|
200
|
+
results_custom_plus_english = detect_profanity(sample_text, language="English", custom_words_list=my_custom_words)
|
|
201
|
+
# results_custom_plus_english will detect "custombadword1" and "asshole"
|
|
202
|
+
# Language will be "Custom + English"
|
|
203
|
+
# [
|
|
204
|
+
# {'Line': 1, 'Column': 15, 'Word': 'custombadword1', 'Language': 'Custom + English'},
|
|
205
|
+
# {'Line': 1, 'Column': 43, 'Word': 'asshole', 'Language': 'Custom + English'}
|
|
206
|
+
# ]
|
|
207
|
+
print(results_custom_plus_english)
|
|
144
208
|
```
|
|
145
209
|
|
|
210
|
+
**3. Loading Custom Profanity Words from a File**
|
|
211
|
+
|
|
212
|
+
ValX provides a helper function to load words from a text file (one word per line, '#' for comments).
|
|
213
|
+
|
|
214
|
+
```python
|
|
215
|
+
from valx import detect_profanity, load_custom_profanity_from_file
|
|
216
|
+
|
|
217
|
+
# Assume 'my_profanity_file.txt' contains:
|
|
218
|
+
# customfileword1
|
|
219
|
+
# # this is a comment
|
|
220
|
+
# customfileword2
|
|
221
|
+
|
|
222
|
+
custom_words_from_file = load_custom_profanity_from_file("my_profanity_file.txt")
|
|
223
|
+
# custom_words_from_file will be: ['customfileword1', 'customfileword2']
|
|
224
|
+
|
|
225
|
+
sample_text_for_file = ["Text with customfileword1 and built-in shit."]
|
|
226
|
+
|
|
227
|
+
# Use file-loaded list with English built-in list
|
|
228
|
+
results_file_plus_english = detect_profanity(
|
|
229
|
+
sample_text_for_file,
|
|
230
|
+
language="English",
|
|
231
|
+
custom_words_list=custom_words_from_file
|
|
232
|
+
)
|
|
233
|
+
# Detects "customfileword1" and "shit", Language: "Custom + English"
|
|
234
|
+
print(results_file_plus_english)
|
|
235
|
+
|
|
236
|
+
# Use file-loaded list ONLY
|
|
237
|
+
results_file_only = detect_profanity(
|
|
238
|
+
sample_text_for_file,
|
|
239
|
+
language=None, # Important: set language to None
|
|
240
|
+
custom_words_list=custom_words_from_file
|
|
241
|
+
)
|
|
242
|
+
# Detects only "customfileword1", Language: "Custom"
|
|
243
|
+
print(results_file_only)
|
|
244
|
+
```
|
|
245
|
+
|
|
246
|
+
**Output Format for `detect_profanity`**
|
|
247
|
+
|
|
248
|
+
The `detect_profanity` function returns a list of dictionaries. Each dictionary includes:
|
|
249
|
+
- `"Line"`: The line number (1-indexed).
|
|
250
|
+
- `"Column"`: The column number (1-indexed) where the profanity starts.
|
|
251
|
+
- `"Word"`: The detected profanity word.
|
|
252
|
+
- `"Language"`: Indicates the source of the word list:
|
|
253
|
+
- `<LanguageName>` (e.g., "English"): If only a built-in language list was used.
|
|
254
|
+
- `"Custom"`: If `language=None` and only a `custom_words_list` was used.
|
|
255
|
+
- `"Custom + <LanguageName>"` (e.g., "Custom + English"): If both a built-in list and `custom_words_list` were used.
|
|
256
|
+
- `"Custom + All"`: If `language='All'` and `custom_words_list` were used.
|
|
257
|
+
|
|
258
|
+
|
|
259
|
+
**4. Removing Profanity**
|
|
260
|
+
|
|
261
|
+
`remove_profanity` works similarly, accepting `language` and `custom_words_list` parameters.
|
|
262
|
+
|
|
263
|
+
```python
|
|
264
|
+
from valx import remove_profanity, load_custom_profanity_from_file
|
|
265
|
+
|
|
266
|
+
sample_text = ["This is fuck, custombadword1, and text with customfileword1."]
|
|
267
|
+
my_custom_words = ["custombadword1"]
|
|
268
|
+
custom_words_from_file = load_custom_profanity_from_file("my_profanity_file.txt") # Assuming it contains 'customfileword1'
|
|
269
|
+
|
|
270
|
+
# Remove profanity using English built-in + my_custom_words + custom_words_from_file
|
|
271
|
+
all_custom_words = list(set(my_custom_words + custom_words_from_file)) # Combine and unique
|
|
272
|
+
|
|
273
|
+
cleaned_text = remove_profanity(
|
|
274
|
+
sample_text,
|
|
275
|
+
output_file="cleaned_output.txt", # Optional: saves to file
|
|
276
|
+
language="English",
|
|
277
|
+
custom_words_list=all_custom_words
|
|
278
|
+
)
|
|
279
|
+
# cleaned_text will have "fuck", "custombadword1", and "customfileword1" replaced with "bad word".
|
|
280
|
+
# e.g., ["This is bad word, bad word, and text with bad word."]
|
|
281
|
+
print(cleaned_text)
|
|
282
|
+
```
|
|
283
|
+
|
|
284
|
+
The `load_profanity_words` function (used internally) also accepts `language` and `custom_words_list` if you need direct access to the word lists.
|
|
285
|
+
|
|
146
286
|
### Detect Sensitive Information
|
|
147
287
|
|
|
148
288
|
```python
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
scikit-learn
|
valx-0.2.3/README.md
DELETED
|
@@ -1,184 +0,0 @@
|
|
|
1
|
-
# ValX
|
|
2
|
-

|
|
3
|
-
[](https://github.com/infinitode/valx)
|
|
4
|
-

|
|
5
|
-

|
|
6
|
-

|
|
7
|
-
|
|
8
|
-
An open-source Python library for data cleaning tasks. It includes functions for profanity detection, and removal, and detection and removal of personal information. Also includes hate speech and offensive language detection and removal, using AI.
|
|
9
|
-
|
|
10
|
-
> [!IMPORTANT]
|
|
11
|
-
> Please downgrade to `numpy` version `1.26.4`. Our ValX **DecisionTreeClassifier** AI model, relies on lower versions of `numpy`, because it was trained on these versions.
|
|
12
|
-
> For more information see: https://techoverflow.net/2024/07/23/how-to-fix-numpy-dtype-size-changed-may-indicate-binary-incompatibility-expected-96-from-c-header-got-88-from-pyobject/
|
|
13
|
-
|
|
14
|
-
## Changes in 0.2.3
|
|
15
|
-
|
|
16
|
-
We have introduced a new optional `info_type` parameter into our `detect_sensitive_information`, and `remove_sensitive_information` functions, to allow you to have fine-grained control over what sensitive information you want to detect or remove.
|
|
17
|
-
|
|
18
|
-
Also introduced more detection patterns for other types of sensitive information, including:
|
|
19
|
-
- `"iban"`: International Bank Account Number.
|
|
20
|
-
- `"mrn"`: Medical Record Number (may not work correctly, depending on provider and country).
|
|
21
|
-
- `"icd10"`: International Classification of Diseases, Tenth Revision.
|
|
22
|
-
- `"geo_coords"`: Geo-coordinates (latitude and longitude in decimal degrees format).
|
|
23
|
-
- `"username"`: Username handles (@username).
|
|
24
|
-
- `"file_path"`: File paths (general patterns for both Windows and Unix paths).
|
|
25
|
-
- `"bitcoin_wallet"`: Cryptocurrency wallet address.
|
|
26
|
-
- `"ethereum_wallet"`: Cryptocurrency wallet addresses.
|
|
27
|
-
|
|
28
|
-
## Changes in 0.2.2
|
|
29
|
-
|
|
30
|
-
We have refactored and changed the `detect_profanity` function:
|
|
31
|
-
- Removed unnecessary printing
|
|
32
|
-
- Now returns more information about each found profanity, including `Line`, `Column`, `Word`, and `Language`.
|
|
33
|
-
|
|
34
|
-
> [!NOTE]
|
|
35
|
-
> You can view [ValX's package documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more information on changes.
|
|
36
|
-
|
|
37
|
-
## Changes in 0.2.1
|
|
38
|
-
|
|
39
|
-
Using the AI models in ValX, you can now automatically remove hate speech, or offensive speech from your text data, without needing to run detection and write your own custom implementation method.
|
|
40
|
-
|
|
41
|
-
## Installation
|
|
42
|
-
|
|
43
|
-
You can install ValX using pip:
|
|
44
|
-
|
|
45
|
-
```bash
|
|
46
|
-
pip install valx
|
|
47
|
-
```
|
|
48
|
-
|
|
49
|
-
## Supported Python Versions
|
|
50
|
-
|
|
51
|
-
ValX supports the following Python versions:
|
|
52
|
-
|
|
53
|
-
- Python 3.6
|
|
54
|
-
- Python 3.7
|
|
55
|
-
- Python 3.8
|
|
56
|
-
- Python 3.9
|
|
57
|
-
- Python 3.10
|
|
58
|
-
- Python 3.11/Later (Preferred)
|
|
59
|
-
|
|
60
|
-
Please ensure that you have one of these Python versions installed before using ValX. ValX may not work as expected on lower versions of Python than the supported.
|
|
61
|
-
|
|
62
|
-
## Features
|
|
63
|
-
|
|
64
|
-
- **Profanity Detection**: Detect profane and NSFW words or terms.
|
|
65
|
-
- **Remove Profanity**: Remove profane and NSFW words or terms.
|
|
66
|
-
- **Detect Sensitive Information**: Detect sensitive information in text data.
|
|
67
|
-
- **Remove Sensitive Information**: Remove sensitive information from text data.
|
|
68
|
-
- **Detect Hate Speech**: Detect hate speech or offensive speech in text, using AI.
|
|
69
|
-
- **Remove Hate Speech**: Remove hate speech or offensive speech in text, using AI.
|
|
70
|
-
|
|
71
|
-
### List of supported languages for profanity detection and removal
|
|
72
|
-
Below is a complete list of all the available supported languages for ValX's profanity detection and removal functions which are valid values for `language`:
|
|
73
|
-
|
|
74
|
-
- **All**
|
|
75
|
-
- Arabic
|
|
76
|
-
- Czech
|
|
77
|
-
- Danish
|
|
78
|
-
- German
|
|
79
|
-
- English
|
|
80
|
-
- Esperanto
|
|
81
|
-
- Persian
|
|
82
|
-
- Finnish
|
|
83
|
-
- Filipino
|
|
84
|
-
- French
|
|
85
|
-
- French (CA)
|
|
86
|
-
- Hindi
|
|
87
|
-
- Hungarian
|
|
88
|
-
- Italian
|
|
89
|
-
- Japanese
|
|
90
|
-
- Kabyle
|
|
91
|
-
- Korean
|
|
92
|
-
- Dutch
|
|
93
|
-
- Norwegian
|
|
94
|
-
- Polish
|
|
95
|
-
- Portuguese
|
|
96
|
-
- Russian
|
|
97
|
-
- Swedish
|
|
98
|
-
- Thai
|
|
99
|
-
- Klingon
|
|
100
|
-
- Turkish
|
|
101
|
-
- Chinese
|
|
102
|
-
|
|
103
|
-
## Usage
|
|
104
|
-
|
|
105
|
-
### Detect Profanity
|
|
106
|
-
|
|
107
|
-
```python
|
|
108
|
-
from valx import detect_profanity
|
|
109
|
-
|
|
110
|
-
# Detect profanity
|
|
111
|
-
results = detect_profanity(sample_text, language='English')
|
|
112
|
-
print("Profanity Evaluation Results", results)
|
|
113
|
-
```
|
|
114
|
-
|
|
115
|
-
### Remove Profanity
|
|
116
|
-
|
|
117
|
-
```python
|
|
118
|
-
from valx import remove_profanity
|
|
119
|
-
|
|
120
|
-
# Remove profanity
|
|
121
|
-
removed = remove_profanity(sample_text, "text_cleaned.txt", language="English")
|
|
122
|
-
```
|
|
123
|
-
|
|
124
|
-
### Detect Sensitive Information
|
|
125
|
-
|
|
126
|
-
```python
|
|
127
|
-
from valx import detect_sensitive_information
|
|
128
|
-
|
|
129
|
-
# Detect sensitive information
|
|
130
|
-
detected_sensitive_info = detect_sensitive_information(sample_text)
|
|
131
|
-
```
|
|
132
|
-
|
|
133
|
-
> [!NOTE]
|
|
134
|
-
> We have updated this function, and it now includes an optional argument for `info_type`, which can be used to detect only specific types of sensitive information. It was also added to `remove_sensitive_information`.
|
|
135
|
-
|
|
136
|
-
### Remove Sensitive Information
|
|
137
|
-
|
|
138
|
-
```python
|
|
139
|
-
from valx import remove_sensitive_information
|
|
140
|
-
|
|
141
|
-
# Remove sensitive information
|
|
142
|
-
cleaned_text = remove_sensitive_information(sample_text2)
|
|
143
|
-
```
|
|
144
|
-
|
|
145
|
-
### Detect Hate Speech And Offensive Language
|
|
146
|
-
|
|
147
|
-
```python
|
|
148
|
-
from valx import detect_hate_speech
|
|
149
|
-
|
|
150
|
-
# Detect hate speech or offensive language
|
|
151
|
-
outcome_of_detection = detect_hate_speech("You are stupid.")
|
|
152
|
-
```
|
|
153
|
-
|
|
154
|
-
> [!IMPORTANT]
|
|
155
|
-
> The model's possible outputs are:
|
|
156
|
-
> - `['Hate Speech']`: The text was flagged and contained hate speech.
|
|
157
|
-
> - `['Offensive Speech']`: The text was flagged and contained offensive speech.
|
|
158
|
-
> - `['No Hate and Offensive Speech']`: The text was not flagged for any hate speech or offensive speech.
|
|
159
|
-
|
|
160
|
-
> [!NOTE]
|
|
161
|
-
> See our [official documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more examples on how to use **ValX**.
|
|
162
|
-
|
|
163
|
-
## Contributing
|
|
164
|
-
|
|
165
|
-
Contributions are welcome! If you encounter any issues, have suggestions, or want to contribute to ValX, please open an issue or submit a pull request on [GitHub](https://github.com/infinitode/valx).
|
|
166
|
-
|
|
167
|
-
## License
|
|
168
|
-
|
|
169
|
-
ValX is released under the terms of the **MIT License (Modified)**. Please see the [LICENSE](https://github.com/infinitode/valx/blob/main/LICENSE) file for the full text.
|
|
170
|
-
|
|
171
|
-
### Derived licenses
|
|
172
|
-
---
|
|
173
|
-
ValX uses data from this GitHub repository:
|
|
174
|
-
https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words/
|
|
175
|
-
© 2012-2020 Shutterstock, Inc.
|
|
176
|
-
|
|
177
|
-
Creative Commons Attribution 4.0 International License:
|
|
178
|
-
https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words/blob/master/LICENSE
|
|
179
|
-
|
|
180
|
-
---
|
|
181
|
-
|
|
182
|
-
**Modified License Clause**
|
|
183
|
-
|
|
184
|
-
The modified license clause grants users the permission to make derivative works based on the ValX software. However, it requires any substantial changes to the software to be clearly distinguished from the original work and distributed under a different name.
|
|
@@ -1 +0,0 @@
|
|
|
1
|
-
scikit-learn==1.2.2
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|