valx 0.2.1__tar.gz → 0.2.2__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {valx-0.2.1/valx.egg-info → valx-0.2.2}/PKG-INFO +57 -4
- {valx-0.2.1 → valx-0.2.2}/README.md +56 -3
- {valx-0.2.1 → valx-0.2.2}/setup.py +1 -1
- {valx-0.2.1 → valx-0.2.2}/valx/__init__.py +17 -10
- {valx-0.2.1 → valx-0.2.2/valx.egg-info}/PKG-INFO +57 -4
- {valx-0.2.1 → valx-0.2.2}/LICENSE +0 -0
- {valx-0.2.1 → valx-0.2.2}/setup.cfg +0 -0
- {valx-0.2.1 → valx-0.2.2}/valx/models/count_vectorizer.sav +0 -0
- {valx-0.2.1 → valx-0.2.2}/valx/models/decision_tree_model.sav +0 -0
- {valx-0.2.1 → valx-0.2.2}/valx.egg-info/SOURCES.txt +0 -0
- {valx-0.2.1 → valx-0.2.2}/valx.egg-info/dependency_links.txt +0 -0
- {valx-0.2.1 → valx-0.2.2}/valx.egg-info/requires.txt +0 -0
- {valx-0.2.1 → valx-0.2.2}/valx.egg-info/top_level.txt +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.1
|
|
2
2
|
Name: valx
|
|
3
|
-
Version: 0.2.
|
|
3
|
+
Version: 0.2.2
|
|
4
4
|
Summary: An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.
|
|
5
5
|
Home-page: https://github.com/infinitode/valx
|
|
6
6
|
Author: Infinitode Pty Ltd
|
|
@@ -27,7 +27,20 @@ Requires-Dist: scikit-learn==1.2.2
|
|
|
27
27
|

|
|
28
28
|

|
|
29
29
|
|
|
30
|
-
An open-source Python library for data cleaning tasks. Includes profanity detection, and removal.
|
|
30
|
+
An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. It also now provides detection and removal of personal information. Now includes hate speech and offensive language detection using AI.
|
|
31
|
+
|
|
32
|
+
> [!IMPORTANT]
|
|
33
|
+
> Please downgrade to `numpy` version `1.26.4`. Our ValX **DecisionTreeClassifier** AI model, relies on lower versions of `numpy`, because it was trained on these versions.
|
|
34
|
+
> For more information see: https://techoverflow.net/2024/07/23/how-to-fix-numpy-dtype-size-changed-may-indicate-binary-incompatibility-expected-96-from-c-header-got-88-from-pyobject/
|
|
35
|
+
|
|
36
|
+
## Changes in 0.2.2
|
|
37
|
+
|
|
38
|
+
We have refactored and changed the `detect_profanity` function:
|
|
39
|
+
- Removed unnecessary printing
|
|
40
|
+
- Now returns more information about each found profanity, including `Line`, `Column`, `Word`, and `Language`.
|
|
41
|
+
|
|
42
|
+
> [!NOTE]
|
|
43
|
+
> You can view [ValX's package documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more information on changes.
|
|
31
44
|
|
|
32
45
|
## Changes in 0.2.1
|
|
33
46
|
|
|
@@ -63,6 +76,38 @@ Please ensure that you have one of these Python versions installed before using
|
|
|
63
76
|
- **Detect Hate Speech**: Detect hate speech or offensive speech in text, using AI.
|
|
64
77
|
- **Remove Hate Speech**: Remove hate speech or offensive speech in text, using AI.
|
|
65
78
|
|
|
79
|
+
### List of supported languages for profanity detection and removal
|
|
80
|
+
Below is a complete list of all the available supported languages for ValX's profanity detection and removal functions which are valid values for `language`:
|
|
81
|
+
|
|
82
|
+
- **All**
|
|
83
|
+
- Arabic
|
|
84
|
+
- Czech
|
|
85
|
+
- Danish
|
|
86
|
+
- German
|
|
87
|
+
- English
|
|
88
|
+
- Esperanto
|
|
89
|
+
- Persian
|
|
90
|
+
- Finnish
|
|
91
|
+
- Filipino
|
|
92
|
+
- French
|
|
93
|
+
- French (CA)
|
|
94
|
+
- Hindi
|
|
95
|
+
- Hungarian
|
|
96
|
+
- Italian
|
|
97
|
+
- Japanese
|
|
98
|
+
- Kabyle
|
|
99
|
+
- Korean
|
|
100
|
+
- Dutch
|
|
101
|
+
- Norwegian
|
|
102
|
+
- Polish
|
|
103
|
+
- Portuguese
|
|
104
|
+
- Russian
|
|
105
|
+
- Swedish
|
|
106
|
+
- Thai
|
|
107
|
+
- Klingon
|
|
108
|
+
- Turkish
|
|
109
|
+
- Chinese
|
|
110
|
+
|
|
66
111
|
## Usage
|
|
67
112
|
|
|
68
113
|
### Detect Profanity
|
|
@@ -71,7 +116,8 @@ Please ensure that you have one of these Python versions installed before using
|
|
|
71
116
|
from valx import detect_profanity
|
|
72
117
|
|
|
73
118
|
# Detect profanity
|
|
74
|
-
|
|
119
|
+
results = detect_profanity(sample_text, language='English')
|
|
120
|
+
print("Profanity Evaluation Results", results)
|
|
75
121
|
```
|
|
76
122
|
|
|
77
123
|
### Remove Profanity
|
|
@@ -110,7 +156,14 @@ from valx import detect_hate_speech
|
|
|
110
156
|
outcome_of_detection = detect_hate_speech("You are stupid.")
|
|
111
157
|
```
|
|
112
158
|
|
|
113
|
-
>
|
|
159
|
+
> [!IMPORTANT]
|
|
160
|
+
> The model's possible outputs are:
|
|
161
|
+
> - `['Hate Speech']`: The text was flagged and contained hate speech.
|
|
162
|
+
> - `['Offensive Speech']`: The text was flagged and contained offensive speech.
|
|
163
|
+
> - `['No Hate and Offensive Speech']`: The text was not flagged for any hate speech or offensive speech.
|
|
164
|
+
|
|
165
|
+
> [!NOTE]
|
|
166
|
+
> See our [official documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more examples on how to use **ValX**.
|
|
114
167
|
|
|
115
168
|
## Contributing
|
|
116
169
|
|
|
@@ -5,7 +5,20 @@
|
|
|
5
5
|

|
|
6
6
|

|
|
7
7
|
|
|
8
|
-
An open-source Python library for data cleaning tasks. Includes profanity detection, and removal.
|
|
8
|
+
An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. It also now provides detection and removal of personal information. Now includes hate speech and offensive language detection using AI.
|
|
9
|
+
|
|
10
|
+
> [!IMPORTANT]
|
|
11
|
+
> Please downgrade to `numpy` version `1.26.4`. Our ValX **DecisionTreeClassifier** AI model, relies on lower versions of `numpy`, because it was trained on these versions.
|
|
12
|
+
> For more information see: https://techoverflow.net/2024/07/23/how-to-fix-numpy-dtype-size-changed-may-indicate-binary-incompatibility-expected-96-from-c-header-got-88-from-pyobject/
|
|
13
|
+
|
|
14
|
+
## Changes in 0.2.2
|
|
15
|
+
|
|
16
|
+
We have refactored and changed the `detect_profanity` function:
|
|
17
|
+
- Removed unnecessary printing
|
|
18
|
+
- Now returns more information about each found profanity, including `Line`, `Column`, `Word`, and `Language`.
|
|
19
|
+
|
|
20
|
+
> [!NOTE]
|
|
21
|
+
> You can view [ValX's package documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more information on changes.
|
|
9
22
|
|
|
10
23
|
## Changes in 0.2.1
|
|
11
24
|
|
|
@@ -41,6 +54,38 @@ Please ensure that you have one of these Python versions installed before using
|
|
|
41
54
|
- **Detect Hate Speech**: Detect hate speech or offensive speech in text, using AI.
|
|
42
55
|
- **Remove Hate Speech**: Remove hate speech or offensive speech in text, using AI.
|
|
43
56
|
|
|
57
|
+
### List of supported languages for profanity detection and removal
|
|
58
|
+
Below is a complete list of all the available supported languages for ValX's profanity detection and removal functions which are valid values for `language`:
|
|
59
|
+
|
|
60
|
+
- **All**
|
|
61
|
+
- Arabic
|
|
62
|
+
- Czech
|
|
63
|
+
- Danish
|
|
64
|
+
- German
|
|
65
|
+
- English
|
|
66
|
+
- Esperanto
|
|
67
|
+
- Persian
|
|
68
|
+
- Finnish
|
|
69
|
+
- Filipino
|
|
70
|
+
- French
|
|
71
|
+
- French (CA)
|
|
72
|
+
- Hindi
|
|
73
|
+
- Hungarian
|
|
74
|
+
- Italian
|
|
75
|
+
- Japanese
|
|
76
|
+
- Kabyle
|
|
77
|
+
- Korean
|
|
78
|
+
- Dutch
|
|
79
|
+
- Norwegian
|
|
80
|
+
- Polish
|
|
81
|
+
- Portuguese
|
|
82
|
+
- Russian
|
|
83
|
+
- Swedish
|
|
84
|
+
- Thai
|
|
85
|
+
- Klingon
|
|
86
|
+
- Turkish
|
|
87
|
+
- Chinese
|
|
88
|
+
|
|
44
89
|
## Usage
|
|
45
90
|
|
|
46
91
|
### Detect Profanity
|
|
@@ -49,7 +94,8 @@ Please ensure that you have one of these Python versions installed before using
|
|
|
49
94
|
from valx import detect_profanity
|
|
50
95
|
|
|
51
96
|
# Detect profanity
|
|
52
|
-
|
|
97
|
+
results = detect_profanity(sample_text, language='English')
|
|
98
|
+
print("Profanity Evaluation Results", results)
|
|
53
99
|
```
|
|
54
100
|
|
|
55
101
|
### Remove Profanity
|
|
@@ -88,7 +134,14 @@ from valx import detect_hate_speech
|
|
|
88
134
|
outcome_of_detection = detect_hate_speech("You are stupid.")
|
|
89
135
|
```
|
|
90
136
|
|
|
91
|
-
>
|
|
137
|
+
> [!IMPORTANT]
|
|
138
|
+
> The model's possible outputs are:
|
|
139
|
+
> - `['Hate Speech']`: The text was flagged and contained hate speech.
|
|
140
|
+
> - `['Offensive Speech']`: The text was flagged and contained offensive speech.
|
|
141
|
+
> - `['No Hate and Offensive Speech']`: The text was not flagged for any hate speech or offensive speech.
|
|
142
|
+
|
|
143
|
+
> [!NOTE]
|
|
144
|
+
> See our [official documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more examples on how to use **ValX**.
|
|
92
145
|
|
|
93
146
|
## Contributing
|
|
94
147
|
|
|
@@ -5,7 +5,7 @@ with open("README.md", "r", encoding="utf-8") as fh:
|
|
|
5
5
|
|
|
6
6
|
setup(
|
|
7
7
|
name='valx',
|
|
8
|
-
version='0.2.
|
|
8
|
+
version='0.2.2',
|
|
9
9
|
author='Infinitode Pty Ltd',
|
|
10
10
|
author_email='infinitode.ltd@gmail.com',
|
|
11
11
|
description='An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.',
|
|
@@ -2754,11 +2754,14 @@ def detect_profanity(text_data, language='English'):
|
|
|
2754
2754
|
|
|
2755
2755
|
Args:
|
|
2756
2756
|
text_data (list): A list of strings representing the text data to analyze.
|
|
2757
|
-
language (str): The language
|
|
2758
|
-
allowed_languages (list): A list of languages to allow for detection. If provided, only profanity words for these languages will be considered.
|
|
2759
|
-
|
|
2757
|
+
language (str): The language used to detect profanity. Defaults to 'English'. Available languages include: All, Arabic, Czech, Danish, German, English, Esperanto, Persian, Finnish, Filipino, French, French (CA), Hindi, Hungarian, Italian, Japanese, Kabyle, Korean, Dutch, Norwegian, Polish, Portuguese, Russian, Swedish, Thai, Klingon, Turkish, Chinese.
|
|
2760
2758
|
Returns:
|
|
2761
|
-
|
|
2759
|
+
list: A list of dictionaries where each dictionary represents a detected instance of profanity.
|
|
2760
|
+
Each dictionary contains the following keys:
|
|
2761
|
+
- "Line" (int): The line number where the profanity was detected.
|
|
2762
|
+
- "Column" (int): The column number (position in the line) where the profanity starts.
|
|
2763
|
+
- "Word" (str): The detected profanity word.
|
|
2764
|
+
- "Language" (str): The language in which the profanity was detected.
|
|
2762
2765
|
"""
|
|
2763
2766
|
profanity_keywords = load_profanity_words(language)
|
|
2764
2767
|
detected = set() # Use a set to store unique occurrences
|
|
@@ -2774,12 +2777,19 @@ def detect_profanity(text_data, language='English'):
|
|
|
2774
2777
|
for match in matches:
|
|
2775
2778
|
detected.add((i + 1, match.start() + 1, language, profanity)) # Add to set
|
|
2776
2779
|
|
|
2780
|
+
detections = [] # Initialize detections as a list
|
|
2781
|
+
|
|
2777
2782
|
if detected:
|
|
2778
|
-
print(f"Profanity detected:")
|
|
2779
2783
|
for line_num, col_num, lang, word in detected:
|
|
2780
|
-
|
|
2784
|
+
detection_info = {
|
|
2785
|
+
"Line": line_num,
|
|
2786
|
+
"Column": col_num,
|
|
2787
|
+
"Word": word,
|
|
2788
|
+
"Language": lang
|
|
2789
|
+
}
|
|
2790
|
+
detections.append(detection_info) # Append each detection to the list
|
|
2781
2791
|
|
|
2782
|
-
return
|
|
2792
|
+
return detections
|
|
2783
2793
|
|
|
2784
2794
|
|
|
2785
2795
|
def remove_profanity(text_data, output_file=None, language='English'):
|
|
@@ -2927,9 +2937,6 @@ def detect_hate_speech(text):
|
|
|
2927
2937
|
Returns:
|
|
2928
2938
|
list of str: A list of strings representing the outcome of the detection.
|
|
2929
2939
|
"""
|
|
2930
|
-
# if isinstance(text, str):
|
|
2931
|
-
# text = [text]
|
|
2932
|
-
|
|
2933
2940
|
text = cv.transform([text]).toarray()
|
|
2934
2941
|
return model.predict((text))
|
|
2935
2942
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.1
|
|
2
2
|
Name: valx
|
|
3
|
-
Version: 0.2.
|
|
3
|
+
Version: 0.2.2
|
|
4
4
|
Summary: An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.
|
|
5
5
|
Home-page: https://github.com/infinitode/valx
|
|
6
6
|
Author: Infinitode Pty Ltd
|
|
@@ -27,7 +27,20 @@ Requires-Dist: scikit-learn==1.2.2
|
|
|
27
27
|

|
|
28
28
|

|
|
29
29
|
|
|
30
|
-
An open-source Python library for data cleaning tasks. Includes profanity detection, and removal.
|
|
30
|
+
An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. It also now provides detection and removal of personal information. Now includes hate speech and offensive language detection using AI.
|
|
31
|
+
|
|
32
|
+
> [!IMPORTANT]
|
|
33
|
+
> Please downgrade to `numpy` version `1.26.4`. Our ValX **DecisionTreeClassifier** AI model, relies on lower versions of `numpy`, because it was trained on these versions.
|
|
34
|
+
> For more information see: https://techoverflow.net/2024/07/23/how-to-fix-numpy-dtype-size-changed-may-indicate-binary-incompatibility-expected-96-from-c-header-got-88-from-pyobject/
|
|
35
|
+
|
|
36
|
+
## Changes in 0.2.2
|
|
37
|
+
|
|
38
|
+
We have refactored and changed the `detect_profanity` function:
|
|
39
|
+
- Removed unnecessary printing
|
|
40
|
+
- Now returns more information about each found profanity, including `Line`, `Column`, `Word`, and `Language`.
|
|
41
|
+
|
|
42
|
+
> [!NOTE]
|
|
43
|
+
> You can view [ValX's package documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more information on changes.
|
|
31
44
|
|
|
32
45
|
## Changes in 0.2.1
|
|
33
46
|
|
|
@@ -63,6 +76,38 @@ Please ensure that you have one of these Python versions installed before using
|
|
|
63
76
|
- **Detect Hate Speech**: Detect hate speech or offensive speech in text, using AI.
|
|
64
77
|
- **Remove Hate Speech**: Remove hate speech or offensive speech in text, using AI.
|
|
65
78
|
|
|
79
|
+
### List of supported languages for profanity detection and removal
|
|
80
|
+
Below is a complete list of all the available supported languages for ValX's profanity detection and removal functions which are valid values for `language`:
|
|
81
|
+
|
|
82
|
+
- **All**
|
|
83
|
+
- Arabic
|
|
84
|
+
- Czech
|
|
85
|
+
- Danish
|
|
86
|
+
- German
|
|
87
|
+
- English
|
|
88
|
+
- Esperanto
|
|
89
|
+
- Persian
|
|
90
|
+
- Finnish
|
|
91
|
+
- Filipino
|
|
92
|
+
- French
|
|
93
|
+
- French (CA)
|
|
94
|
+
- Hindi
|
|
95
|
+
- Hungarian
|
|
96
|
+
- Italian
|
|
97
|
+
- Japanese
|
|
98
|
+
- Kabyle
|
|
99
|
+
- Korean
|
|
100
|
+
- Dutch
|
|
101
|
+
- Norwegian
|
|
102
|
+
- Polish
|
|
103
|
+
- Portuguese
|
|
104
|
+
- Russian
|
|
105
|
+
- Swedish
|
|
106
|
+
- Thai
|
|
107
|
+
- Klingon
|
|
108
|
+
- Turkish
|
|
109
|
+
- Chinese
|
|
110
|
+
|
|
66
111
|
## Usage
|
|
67
112
|
|
|
68
113
|
### Detect Profanity
|
|
@@ -71,7 +116,8 @@ Please ensure that you have one of these Python versions installed before using
|
|
|
71
116
|
from valx import detect_profanity
|
|
72
117
|
|
|
73
118
|
# Detect profanity
|
|
74
|
-
|
|
119
|
+
results = detect_profanity(sample_text, language='English')
|
|
120
|
+
print("Profanity Evaluation Results", results)
|
|
75
121
|
```
|
|
76
122
|
|
|
77
123
|
### Remove Profanity
|
|
@@ -110,7 +156,14 @@ from valx import detect_hate_speech
|
|
|
110
156
|
outcome_of_detection = detect_hate_speech("You are stupid.")
|
|
111
157
|
```
|
|
112
158
|
|
|
113
|
-
>
|
|
159
|
+
> [!IMPORTANT]
|
|
160
|
+
> The model's possible outputs are:
|
|
161
|
+
> - `['Hate Speech']`: The text was flagged and contained hate speech.
|
|
162
|
+
> - `['Offensive Speech']`: The text was flagged and contained offensive speech.
|
|
163
|
+
> - `['No Hate and Offensive Speech']`: The text was not flagged for any hate speech or offensive speech.
|
|
164
|
+
|
|
165
|
+
> [!NOTE]
|
|
166
|
+
> See our [official documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more examples on how to use **ValX**.
|
|
114
167
|
|
|
115
168
|
## Contributing
|
|
116
169
|
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|