valx 0.2.0__tar.gz → 0.2.2__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- valx-0.2.2/PKG-INFO +189 -0
- {valx-0.2.0 → valx-0.2.2}/README.md +57 -4
- {valx-0.2.0 → valx-0.2.2}/setup.py +6 -3
- {valx-0.2.0 → valx-0.2.2}/valx/__init__.py +17 -10
- valx-0.2.2/valx.egg-info/PKG-INFO +189 -0
- valx-0.2.0/PKG-INFO +0 -23
- valx-0.2.0/valx.egg-info/PKG-INFO +0 -23
- {valx-0.2.0 → valx-0.2.2}/LICENSE +0 -0
- {valx-0.2.0 → valx-0.2.2}/setup.cfg +0 -0
- {valx-0.2.0 → valx-0.2.2}/valx/models/count_vectorizer.sav +0 -0
- {valx-0.2.0 → valx-0.2.2}/valx/models/decision_tree_model.sav +0 -0
- {valx-0.2.0 → valx-0.2.2}/valx.egg-info/SOURCES.txt +0 -0
- {valx-0.2.0 → valx-0.2.2}/valx.egg-info/dependency_links.txt +0 -0
- {valx-0.2.0 → valx-0.2.2}/valx.egg-info/requires.txt +0 -0
- {valx-0.2.0 → valx-0.2.2}/valx.egg-info/top_level.txt +0 -0
valx-0.2.2/PKG-INFO
ADDED
|
@@ -0,0 +1,189 @@
|
|
|
1
|
+
Metadata-Version: 2.1
|
|
2
|
+
Name: valx
|
|
3
|
+
Version: 0.2.2
|
|
4
|
+
Summary: An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.
|
|
5
|
+
Home-page: https://github.com/infinitode/valx
|
|
6
|
+
Author: Infinitode Pty Ltd
|
|
7
|
+
Author-email: infinitode.ltd@gmail.com
|
|
8
|
+
Classifier: Development Status :: 3 - Alpha
|
|
9
|
+
Classifier: Intended Audience :: Developers
|
|
10
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
11
|
+
Classifier: Programming Language :: Python :: 3.6
|
|
12
|
+
Classifier: Programming Language :: Python :: 3.7
|
|
13
|
+
Classifier: Programming Language :: Python :: 3.8
|
|
14
|
+
Classifier: Programming Language :: Python :: 3.9
|
|
15
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
18
|
+
Requires-Python: >=3.6
|
|
19
|
+
Description-Content-Type: text/markdown
|
|
20
|
+
License-File: LICENSE
|
|
21
|
+
Requires-Dist: scikit-learn==1.2.2
|
|
22
|
+
|
|
23
|
+
# ValX
|
|
24
|
+

|
|
25
|
+
[](https://github.com/infinitode/valx)
|
|
26
|
+

|
|
27
|
+

|
|
28
|
+

|
|
29
|
+
|
|
30
|
+
An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. It also now provides detection and removal of personal information. Now includes hate speech and offensive language detection using AI.
|
|
31
|
+
|
|
32
|
+
> [!IMPORTANT]
|
|
33
|
+
> Please downgrade to `numpy` version `1.26.4`. Our ValX **DecisionTreeClassifier** AI model, relies on lower versions of `numpy`, because it was trained on these versions.
|
|
34
|
+
> For more information see: https://techoverflow.net/2024/07/23/how-to-fix-numpy-dtype-size-changed-may-indicate-binary-incompatibility-expected-96-from-c-header-got-88-from-pyobject/
|
|
35
|
+
|
|
36
|
+
## Changes in 0.2.2
|
|
37
|
+
|
|
38
|
+
We have refactored and changed the `detect_profanity` function:
|
|
39
|
+
- Removed unnecessary printing
|
|
40
|
+
- Now returns more information about each found profanity, including `Line`, `Column`, `Word`, and `Language`.
|
|
41
|
+
|
|
42
|
+
> [!NOTE]
|
|
43
|
+
> You can view [ValX's package documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more information on changes.
|
|
44
|
+
|
|
45
|
+
## Changes in 0.2.1
|
|
46
|
+
|
|
47
|
+
Using the AI models in ValX, you can now automatically remove hate speech, or offensive speech from your text data, without needing to run detection and write your own custom implementation method.
|
|
48
|
+
|
|
49
|
+
## Installation
|
|
50
|
+
|
|
51
|
+
You can install ValX using pip:
|
|
52
|
+
|
|
53
|
+
```bash
|
|
54
|
+
pip install valx
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
## Supported Python Versions
|
|
58
|
+
|
|
59
|
+
ValX supports the following Python versions:
|
|
60
|
+
|
|
61
|
+
- Python 3.6
|
|
62
|
+
- Python 3.7
|
|
63
|
+
- Python 3.8
|
|
64
|
+
- Python 3.9
|
|
65
|
+
- Python 3.10
|
|
66
|
+
- Python 3.11/Later (Preferred)
|
|
67
|
+
|
|
68
|
+
Please ensure that you have one of these Python versions installed before using ValX. ValX may not work as expected on lower versions of Python than the supported.
|
|
69
|
+
|
|
70
|
+
## Features
|
|
71
|
+
|
|
72
|
+
- **Profanity Detection**: Detect profane and NSFW words or terms.
|
|
73
|
+
- **Remove Profanity**: Remove profane and NSFW words or terms.
|
|
74
|
+
- **Detect Sensitive Information**: Detect sensitive information in text data.
|
|
75
|
+
- **Remove Sensitive Information**: Remove sensitive information from text data.
|
|
76
|
+
- **Detect Hate Speech**: Detect hate speech or offensive speech in text, using AI.
|
|
77
|
+
- **Remove Hate Speech**: Remove hate speech or offensive speech in text, using AI.
|
|
78
|
+
|
|
79
|
+
### List of supported languages for profanity detection and removal
|
|
80
|
+
Below is a complete list of all the available supported languages for ValX's profanity detection and removal functions which are valid values for `language`:
|
|
81
|
+
|
|
82
|
+
- **All**
|
|
83
|
+
- Arabic
|
|
84
|
+
- Czech
|
|
85
|
+
- Danish
|
|
86
|
+
- German
|
|
87
|
+
- English
|
|
88
|
+
- Esperanto
|
|
89
|
+
- Persian
|
|
90
|
+
- Finnish
|
|
91
|
+
- Filipino
|
|
92
|
+
- French
|
|
93
|
+
- French (CA)
|
|
94
|
+
- Hindi
|
|
95
|
+
- Hungarian
|
|
96
|
+
- Italian
|
|
97
|
+
- Japanese
|
|
98
|
+
- Kabyle
|
|
99
|
+
- Korean
|
|
100
|
+
- Dutch
|
|
101
|
+
- Norwegian
|
|
102
|
+
- Polish
|
|
103
|
+
- Portuguese
|
|
104
|
+
- Russian
|
|
105
|
+
- Swedish
|
|
106
|
+
- Thai
|
|
107
|
+
- Klingon
|
|
108
|
+
- Turkish
|
|
109
|
+
- Chinese
|
|
110
|
+
|
|
111
|
+
## Usage
|
|
112
|
+
|
|
113
|
+
### Detect Profanity
|
|
114
|
+
|
|
115
|
+
```python
|
|
116
|
+
from valx import detect_profanity
|
|
117
|
+
|
|
118
|
+
# Detect profanity
|
|
119
|
+
results = detect_profanity(sample_text, language='English')
|
|
120
|
+
print("Profanity Evaluation Results", results)
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
### Remove Profanity
|
|
124
|
+
|
|
125
|
+
```python
|
|
126
|
+
from valx import remove_profanity
|
|
127
|
+
|
|
128
|
+
# Remove profanity
|
|
129
|
+
removed = remove_profanity(sample_text, "text_cleaned.txt", language="English")
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
### Detect Sensitive Information
|
|
133
|
+
|
|
134
|
+
```python
|
|
135
|
+
from valx import detect_sensitive_information
|
|
136
|
+
|
|
137
|
+
# Detect sensitive information
|
|
138
|
+
detected_sensitive_info = detect_sensitive_information(sample_text)
|
|
139
|
+
```
|
|
140
|
+
|
|
141
|
+
### Remove Sensitive Information
|
|
142
|
+
|
|
143
|
+
```python
|
|
144
|
+
from valx import remove_sensitive_information
|
|
145
|
+
|
|
146
|
+
# Remove sensitive information
|
|
147
|
+
cleaned_text = remove_sensitive_information(sample_text2)
|
|
148
|
+
```
|
|
149
|
+
|
|
150
|
+
### Detect Hate Speech And Offensive Language
|
|
151
|
+
|
|
152
|
+
```python
|
|
153
|
+
from valx import detect_hate_speech
|
|
154
|
+
|
|
155
|
+
# Detect hate speech or offensive language
|
|
156
|
+
outcome_of_detection = detect_hate_speech("You are stupid.")
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
> [!IMPORTANT]
|
|
160
|
+
> The model's possible outputs are:
|
|
161
|
+
> - `['Hate Speech']`: The text was flagged and contained hate speech.
|
|
162
|
+
> - `['Offensive Speech']`: The text was flagged and contained offensive speech.
|
|
163
|
+
> - `['No Hate and Offensive Speech']`: The text was not flagged for any hate speech or offensive speech.
|
|
164
|
+
|
|
165
|
+
> [!NOTE]
|
|
166
|
+
> See our [official documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more examples on how to use **ValX**.
|
|
167
|
+
|
|
168
|
+
## Contributing
|
|
169
|
+
|
|
170
|
+
Contributions are welcome! If you encounter any issues, have suggestions, or want to contribute to ValX, please open an issue or submit a pull request on [GitHub](https://github.com/infinitode/valx).
|
|
171
|
+
|
|
172
|
+
## License
|
|
173
|
+
|
|
174
|
+
ValX is released under the terms of the **MIT License (Modified)**. Please see the [LICENSE](https://github.com/infinitode/valx/blob/main/LICENSE) file for the full text.
|
|
175
|
+
|
|
176
|
+
### Derived licenses
|
|
177
|
+
---
|
|
178
|
+
ValX uses data from this GitHub repository:
|
|
179
|
+
https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words/
|
|
180
|
+
© 2012-2020 Shutterstock, Inc.
|
|
181
|
+
|
|
182
|
+
Creative Commons Attribution 4.0 International License:
|
|
183
|
+
https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words/blob/master/LICENSE
|
|
184
|
+
|
|
185
|
+
---
|
|
186
|
+
|
|
187
|
+
**Modified License Clause**
|
|
188
|
+
|
|
189
|
+
The modified license clause grants users the permission to make derivative works based on the ValX software. However, it requires any substantial changes to the software to be clearly distinguished from the original work and distributed under a different name.
|
|
@@ -5,9 +5,22 @@
|
|
|
5
5
|

|
|
6
6
|

|
|
7
7
|
|
|
8
|
-
An open-source Python library for data cleaning tasks. Includes profanity detection, and removal.
|
|
8
|
+
An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. It also now provides detection and removal of personal information. Now includes hate speech and offensive language detection using AI.
|
|
9
9
|
|
|
10
|
-
|
|
10
|
+
> [!IMPORTANT]
|
|
11
|
+
> Please downgrade to `numpy` version `1.26.4`. Our ValX **DecisionTreeClassifier** AI model, relies on lower versions of `numpy`, because it was trained on these versions.
|
|
12
|
+
> For more information see: https://techoverflow.net/2024/07/23/how-to-fix-numpy-dtype-size-changed-may-indicate-binary-incompatibility-expected-96-from-c-header-got-88-from-pyobject/
|
|
13
|
+
|
|
14
|
+
## Changes in 0.2.2
|
|
15
|
+
|
|
16
|
+
We have refactored and changed the `detect_profanity` function:
|
|
17
|
+
- Removed unnecessary printing
|
|
18
|
+
- Now returns more information about each found profanity, including `Line`, `Column`, `Word`, and `Language`.
|
|
19
|
+
|
|
20
|
+
> [!NOTE]
|
|
21
|
+
> You can view [ValX's package documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more information on changes.
|
|
22
|
+
|
|
23
|
+
## Changes in 0.2.1
|
|
11
24
|
|
|
12
25
|
Using the AI models in ValX, you can now automatically remove hate speech, or offensive speech from your text data, without needing to run detection and write your own custom implementation method.
|
|
13
26
|
|
|
@@ -41,6 +54,38 @@ Please ensure that you have one of these Python versions installed before using
|
|
|
41
54
|
- **Detect Hate Speech**: Detect hate speech or offensive speech in text, using AI.
|
|
42
55
|
- **Remove Hate Speech**: Remove hate speech or offensive speech in text, using AI.
|
|
43
56
|
|
|
57
|
+
### List of supported languages for profanity detection and removal
|
|
58
|
+
Below is a complete list of all the available supported languages for ValX's profanity detection and removal functions which are valid values for `language`:
|
|
59
|
+
|
|
60
|
+
- **All**
|
|
61
|
+
- Arabic
|
|
62
|
+
- Czech
|
|
63
|
+
- Danish
|
|
64
|
+
- German
|
|
65
|
+
- English
|
|
66
|
+
- Esperanto
|
|
67
|
+
- Persian
|
|
68
|
+
- Finnish
|
|
69
|
+
- Filipino
|
|
70
|
+
- French
|
|
71
|
+
- French (CA)
|
|
72
|
+
- Hindi
|
|
73
|
+
- Hungarian
|
|
74
|
+
- Italian
|
|
75
|
+
- Japanese
|
|
76
|
+
- Kabyle
|
|
77
|
+
- Korean
|
|
78
|
+
- Dutch
|
|
79
|
+
- Norwegian
|
|
80
|
+
- Polish
|
|
81
|
+
- Portuguese
|
|
82
|
+
- Russian
|
|
83
|
+
- Swedish
|
|
84
|
+
- Thai
|
|
85
|
+
- Klingon
|
|
86
|
+
- Turkish
|
|
87
|
+
- Chinese
|
|
88
|
+
|
|
44
89
|
## Usage
|
|
45
90
|
|
|
46
91
|
### Detect Profanity
|
|
@@ -49,7 +94,8 @@ Please ensure that you have one of these Python versions installed before using
|
|
|
49
94
|
from valx import detect_profanity
|
|
50
95
|
|
|
51
96
|
# Detect profanity
|
|
52
|
-
|
|
97
|
+
results = detect_profanity(sample_text, language='English')
|
|
98
|
+
print("Profanity Evaluation Results", results)
|
|
53
99
|
```
|
|
54
100
|
|
|
55
101
|
### Remove Profanity
|
|
@@ -88,7 +134,14 @@ from valx import detect_hate_speech
|
|
|
88
134
|
outcome_of_detection = detect_hate_speech("You are stupid.")
|
|
89
135
|
```
|
|
90
136
|
|
|
91
|
-
|
|
137
|
+
> [!IMPORTANT]
|
|
138
|
+
> The model's possible outputs are:
|
|
139
|
+
> - `['Hate Speech']`: The text was flagged and contained hate speech.
|
|
140
|
+
> - `['Offensive Speech']`: The text was flagged and contained offensive speech.
|
|
141
|
+
> - `['No Hate and Offensive Speech']`: The text was not flagged for any hate speech or offensive speech.
|
|
142
|
+
|
|
143
|
+
> [!NOTE]
|
|
144
|
+
> See our [official documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more examples on how to use **ValX**.
|
|
92
145
|
|
|
93
146
|
## Contributing
|
|
94
147
|
|
|
@@ -1,12 +1,15 @@
|
|
|
1
1
|
from setuptools import setup, find_packages
|
|
2
2
|
|
|
3
|
+
with open("README.md", "r", encoding="utf-8") as fh:
|
|
4
|
+
long_description = fh.read()
|
|
5
|
+
|
|
3
6
|
setup(
|
|
4
7
|
name='valx',
|
|
5
|
-
version='0.2.
|
|
8
|
+
version='0.2.2',
|
|
6
9
|
author='Infinitode Pty Ltd',
|
|
7
10
|
author_email='infinitode.ltd@gmail.com',
|
|
8
|
-
description='An open-source Python library for
|
|
9
|
-
long_description=
|
|
11
|
+
description='An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.',
|
|
12
|
+
long_description=long_description,
|
|
10
13
|
long_description_content_type='text/markdown',
|
|
11
14
|
url='https://github.com/infinitode/valx',
|
|
12
15
|
packages=find_packages(),
|
|
@@ -2754,11 +2754,14 @@ def detect_profanity(text_data, language='English'):
|
|
|
2754
2754
|
|
|
2755
2755
|
Args:
|
|
2756
2756
|
text_data (list): A list of strings representing the text data to analyze.
|
|
2757
|
-
language (str): The language
|
|
2758
|
-
allowed_languages (list): A list of languages to allow for detection. If provided, only profanity words for these languages will be considered.
|
|
2759
|
-
|
|
2757
|
+
language (str): The language used to detect profanity. Defaults to 'English'. Available languages include: All, Arabic, Czech, Danish, German, English, Esperanto, Persian, Finnish, Filipino, French, French (CA), Hindi, Hungarian, Italian, Japanese, Kabyle, Korean, Dutch, Norwegian, Polish, Portuguese, Russian, Swedish, Thai, Klingon, Turkish, Chinese.
|
|
2760
2758
|
Returns:
|
|
2761
|
-
|
|
2759
|
+
list: A list of dictionaries where each dictionary represents a detected instance of profanity.
|
|
2760
|
+
Each dictionary contains the following keys:
|
|
2761
|
+
- "Line" (int): The line number where the profanity was detected.
|
|
2762
|
+
- "Column" (int): The column number (position in the line) where the profanity starts.
|
|
2763
|
+
- "Word" (str): The detected profanity word.
|
|
2764
|
+
- "Language" (str): The language in which the profanity was detected.
|
|
2762
2765
|
"""
|
|
2763
2766
|
profanity_keywords = load_profanity_words(language)
|
|
2764
2767
|
detected = set() # Use a set to store unique occurrences
|
|
@@ -2774,12 +2777,19 @@ def detect_profanity(text_data, language='English'):
|
|
|
2774
2777
|
for match in matches:
|
|
2775
2778
|
detected.add((i + 1, match.start() + 1, language, profanity)) # Add to set
|
|
2776
2779
|
|
|
2780
|
+
detections = [] # Initialize detections as a list
|
|
2781
|
+
|
|
2777
2782
|
if detected:
|
|
2778
|
-
print(f"Profanity detected:")
|
|
2779
2783
|
for line_num, col_num, lang, word in detected:
|
|
2780
|
-
|
|
2784
|
+
detection_info = {
|
|
2785
|
+
"Line": line_num,
|
|
2786
|
+
"Column": col_num,
|
|
2787
|
+
"Word": word,
|
|
2788
|
+
"Language": lang
|
|
2789
|
+
}
|
|
2790
|
+
detections.append(detection_info) # Append each detection to the list
|
|
2781
2791
|
|
|
2782
|
-
return
|
|
2792
|
+
return detections
|
|
2783
2793
|
|
|
2784
2794
|
|
|
2785
2795
|
def remove_profanity(text_data, output_file=None, language='English'):
|
|
@@ -2927,9 +2937,6 @@ def detect_hate_speech(text):
|
|
|
2927
2937
|
Returns:
|
|
2928
2938
|
list of str: A list of strings representing the outcome of the detection.
|
|
2929
2939
|
"""
|
|
2930
|
-
# if isinstance(text, str):
|
|
2931
|
-
# text = [text]
|
|
2932
|
-
|
|
2933
2940
|
text = cv.transform([text]).toarray()
|
|
2934
2941
|
return model.predict((text))
|
|
2935
2942
|
|
|
@@ -0,0 +1,189 @@
|
|
|
1
|
+
Metadata-Version: 2.1
|
|
2
|
+
Name: valx
|
|
3
|
+
Version: 0.2.2
|
|
4
|
+
Summary: An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.
|
|
5
|
+
Home-page: https://github.com/infinitode/valx
|
|
6
|
+
Author: Infinitode Pty Ltd
|
|
7
|
+
Author-email: infinitode.ltd@gmail.com
|
|
8
|
+
Classifier: Development Status :: 3 - Alpha
|
|
9
|
+
Classifier: Intended Audience :: Developers
|
|
10
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
11
|
+
Classifier: Programming Language :: Python :: 3.6
|
|
12
|
+
Classifier: Programming Language :: Python :: 3.7
|
|
13
|
+
Classifier: Programming Language :: Python :: 3.8
|
|
14
|
+
Classifier: Programming Language :: Python :: 3.9
|
|
15
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
18
|
+
Requires-Python: >=3.6
|
|
19
|
+
Description-Content-Type: text/markdown
|
|
20
|
+
License-File: LICENSE
|
|
21
|
+
Requires-Dist: scikit-learn==1.2.2
|
|
22
|
+
|
|
23
|
+
# ValX
|
|
24
|
+

|
|
25
|
+
[](https://github.com/infinitode/valx)
|
|
26
|
+

|
|
27
|
+

|
|
28
|
+

|
|
29
|
+
|
|
30
|
+
An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. It also now provides detection and removal of personal information. Now includes hate speech and offensive language detection using AI.
|
|
31
|
+
|
|
32
|
+
> [!IMPORTANT]
|
|
33
|
+
> Please downgrade to `numpy` version `1.26.4`. Our ValX **DecisionTreeClassifier** AI model, relies on lower versions of `numpy`, because it was trained on these versions.
|
|
34
|
+
> For more information see: https://techoverflow.net/2024/07/23/how-to-fix-numpy-dtype-size-changed-may-indicate-binary-incompatibility-expected-96-from-c-header-got-88-from-pyobject/
|
|
35
|
+
|
|
36
|
+
## Changes in 0.2.2
|
|
37
|
+
|
|
38
|
+
We have refactored and changed the `detect_profanity` function:
|
|
39
|
+
- Removed unnecessary printing
|
|
40
|
+
- Now returns more information about each found profanity, including `Line`, `Column`, `Word`, and `Language`.
|
|
41
|
+
|
|
42
|
+
> [!NOTE]
|
|
43
|
+
> You can view [ValX's package documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more information on changes.
|
|
44
|
+
|
|
45
|
+
## Changes in 0.2.1
|
|
46
|
+
|
|
47
|
+
Using the AI models in ValX, you can now automatically remove hate speech, or offensive speech from your text data, without needing to run detection and write your own custom implementation method.
|
|
48
|
+
|
|
49
|
+
## Installation
|
|
50
|
+
|
|
51
|
+
You can install ValX using pip:
|
|
52
|
+
|
|
53
|
+
```bash
|
|
54
|
+
pip install valx
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
## Supported Python Versions
|
|
58
|
+
|
|
59
|
+
ValX supports the following Python versions:
|
|
60
|
+
|
|
61
|
+
- Python 3.6
|
|
62
|
+
- Python 3.7
|
|
63
|
+
- Python 3.8
|
|
64
|
+
- Python 3.9
|
|
65
|
+
- Python 3.10
|
|
66
|
+
- Python 3.11/Later (Preferred)
|
|
67
|
+
|
|
68
|
+
Please ensure that you have one of these Python versions installed before using ValX. ValX may not work as expected on lower versions of Python than the supported.
|
|
69
|
+
|
|
70
|
+
## Features
|
|
71
|
+
|
|
72
|
+
- **Profanity Detection**: Detect profane and NSFW words or terms.
|
|
73
|
+
- **Remove Profanity**: Remove profane and NSFW words or terms.
|
|
74
|
+
- **Detect Sensitive Information**: Detect sensitive information in text data.
|
|
75
|
+
- **Remove Sensitive Information**: Remove sensitive information from text data.
|
|
76
|
+
- **Detect Hate Speech**: Detect hate speech or offensive speech in text, using AI.
|
|
77
|
+
- **Remove Hate Speech**: Remove hate speech or offensive speech in text, using AI.
|
|
78
|
+
|
|
79
|
+
### List of supported languages for profanity detection and removal
|
|
80
|
+
Below is a complete list of all the available supported languages for ValX's profanity detection and removal functions which are valid values for `language`:
|
|
81
|
+
|
|
82
|
+
- **All**
|
|
83
|
+
- Arabic
|
|
84
|
+
- Czech
|
|
85
|
+
- Danish
|
|
86
|
+
- German
|
|
87
|
+
- English
|
|
88
|
+
- Esperanto
|
|
89
|
+
- Persian
|
|
90
|
+
- Finnish
|
|
91
|
+
- Filipino
|
|
92
|
+
- French
|
|
93
|
+
- French (CA)
|
|
94
|
+
- Hindi
|
|
95
|
+
- Hungarian
|
|
96
|
+
- Italian
|
|
97
|
+
- Japanese
|
|
98
|
+
- Kabyle
|
|
99
|
+
- Korean
|
|
100
|
+
- Dutch
|
|
101
|
+
- Norwegian
|
|
102
|
+
- Polish
|
|
103
|
+
- Portuguese
|
|
104
|
+
- Russian
|
|
105
|
+
- Swedish
|
|
106
|
+
- Thai
|
|
107
|
+
- Klingon
|
|
108
|
+
- Turkish
|
|
109
|
+
- Chinese
|
|
110
|
+
|
|
111
|
+
## Usage
|
|
112
|
+
|
|
113
|
+
### Detect Profanity
|
|
114
|
+
|
|
115
|
+
```python
|
|
116
|
+
from valx import detect_profanity
|
|
117
|
+
|
|
118
|
+
# Detect profanity
|
|
119
|
+
results = detect_profanity(sample_text, language='English')
|
|
120
|
+
print("Profanity Evaluation Results", results)
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
### Remove Profanity
|
|
124
|
+
|
|
125
|
+
```python
|
|
126
|
+
from valx import remove_profanity
|
|
127
|
+
|
|
128
|
+
# Remove profanity
|
|
129
|
+
removed = remove_profanity(sample_text, "text_cleaned.txt", language="English")
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
### Detect Sensitive Information
|
|
133
|
+
|
|
134
|
+
```python
|
|
135
|
+
from valx import detect_sensitive_information
|
|
136
|
+
|
|
137
|
+
# Detect sensitive information
|
|
138
|
+
detected_sensitive_info = detect_sensitive_information(sample_text)
|
|
139
|
+
```
|
|
140
|
+
|
|
141
|
+
### Remove Sensitive Information
|
|
142
|
+
|
|
143
|
+
```python
|
|
144
|
+
from valx import remove_sensitive_information
|
|
145
|
+
|
|
146
|
+
# Remove sensitive information
|
|
147
|
+
cleaned_text = remove_sensitive_information(sample_text2)
|
|
148
|
+
```
|
|
149
|
+
|
|
150
|
+
### Detect Hate Speech And Offensive Language
|
|
151
|
+
|
|
152
|
+
```python
|
|
153
|
+
from valx import detect_hate_speech
|
|
154
|
+
|
|
155
|
+
# Detect hate speech or offensive language
|
|
156
|
+
outcome_of_detection = detect_hate_speech("You are stupid.")
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
> [!IMPORTANT]
|
|
160
|
+
> The model's possible outputs are:
|
|
161
|
+
> - `['Hate Speech']`: The text was flagged and contained hate speech.
|
|
162
|
+
> - `['Offensive Speech']`: The text was flagged and contained offensive speech.
|
|
163
|
+
> - `['No Hate and Offensive Speech']`: The text was not flagged for any hate speech or offensive speech.
|
|
164
|
+
|
|
165
|
+
> [!NOTE]
|
|
166
|
+
> See our [official documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more examples on how to use **ValX**.
|
|
167
|
+
|
|
168
|
+
## Contributing
|
|
169
|
+
|
|
170
|
+
Contributions are welcome! If you encounter any issues, have suggestions, or want to contribute to ValX, please open an issue or submit a pull request on [GitHub](https://github.com/infinitode/valx).
|
|
171
|
+
|
|
172
|
+
## License
|
|
173
|
+
|
|
174
|
+
ValX is released under the terms of the **MIT License (Modified)**. Please see the [LICENSE](https://github.com/infinitode/valx/blob/main/LICENSE) file for the full text.
|
|
175
|
+
|
|
176
|
+
### Derived licenses
|
|
177
|
+
---
|
|
178
|
+
ValX uses data from this GitHub repository:
|
|
179
|
+
https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words/
|
|
180
|
+
© 2012-2020 Shutterstock, Inc.
|
|
181
|
+
|
|
182
|
+
Creative Commons Attribution 4.0 International License:
|
|
183
|
+
https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words/blob/master/LICENSE
|
|
184
|
+
|
|
185
|
+
---
|
|
186
|
+
|
|
187
|
+
**Modified License Clause**
|
|
188
|
+
|
|
189
|
+
The modified license clause grants users the permission to make derivative works based on the ValX software. However, it requires any substantial changes to the software to be clearly distinguished from the original work and distributed under a different name.
|
valx-0.2.0/PKG-INFO
DELETED
|
@@ -1,23 +0,0 @@
|
|
|
1
|
-
Metadata-Version: 2.1
|
|
2
|
-
Name: valx
|
|
3
|
-
Version: 0.2.0
|
|
4
|
-
Summary: An open-source Python library for text cleaning tasks.
|
|
5
|
-
Home-page: https://github.com/infinitode/valx
|
|
6
|
-
Author: Infinitode Pty Ltd
|
|
7
|
-
Author-email: infinitode.ltd@gmail.com
|
|
8
|
-
Classifier: Development Status :: 3 - Alpha
|
|
9
|
-
Classifier: Intended Audience :: Developers
|
|
10
|
-
Classifier: License :: OSI Approved :: MIT License
|
|
11
|
-
Classifier: Programming Language :: Python :: 3.6
|
|
12
|
-
Classifier: Programming Language :: Python :: 3.7
|
|
13
|
-
Classifier: Programming Language :: Python :: 3.8
|
|
14
|
-
Classifier: Programming Language :: Python :: 3.9
|
|
15
|
-
Classifier: Programming Language :: Python :: 3.10
|
|
16
|
-
Classifier: Programming Language :: Python :: 3.11
|
|
17
|
-
Classifier: Programming Language :: Python :: 3.12
|
|
18
|
-
Requires-Python: >=3.6
|
|
19
|
-
Description-Content-Type: text/markdown
|
|
20
|
-
License-File: LICENSE
|
|
21
|
-
Requires-Dist: scikit-learn==1.2.2
|
|
22
|
-
|
|
23
|
-
An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.
|
|
@@ -1,23 +0,0 @@
|
|
|
1
|
-
Metadata-Version: 2.1
|
|
2
|
-
Name: valx
|
|
3
|
-
Version: 0.2.0
|
|
4
|
-
Summary: An open-source Python library for text cleaning tasks.
|
|
5
|
-
Home-page: https://github.com/infinitode/valx
|
|
6
|
-
Author: Infinitode Pty Ltd
|
|
7
|
-
Author-email: infinitode.ltd@gmail.com
|
|
8
|
-
Classifier: Development Status :: 3 - Alpha
|
|
9
|
-
Classifier: Intended Audience :: Developers
|
|
10
|
-
Classifier: License :: OSI Approved :: MIT License
|
|
11
|
-
Classifier: Programming Language :: Python :: 3.6
|
|
12
|
-
Classifier: Programming Language :: Python :: 3.7
|
|
13
|
-
Classifier: Programming Language :: Python :: 3.8
|
|
14
|
-
Classifier: Programming Language :: Python :: 3.9
|
|
15
|
-
Classifier: Programming Language :: Python :: 3.10
|
|
16
|
-
Classifier: Programming Language :: Python :: 3.11
|
|
17
|
-
Classifier: Programming Language :: Python :: 3.12
|
|
18
|
-
Requires-Python: >=3.6
|
|
19
|
-
Description-Content-Type: text/markdown
|
|
20
|
-
License-File: LICENSE
|
|
21
|
-
Requires-Dist: scikit-learn==1.2.2
|
|
22
|
-
|
|
23
|
-
An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|