valx 0.2.1__tar.gz → 0.2.2__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.1
2
2
  Name: valx
3
- Version: 0.2.1
3
+ Version: 0.2.2
4
4
  Summary: An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.
5
5
  Home-page: https://github.com/infinitode/valx
6
6
  Author: Infinitode Pty Ltd
@@ -27,7 +27,20 @@ Requires-Dist: scikit-learn==1.2.2
27
27
  ![License Compliance](https://img.shields.io/badge/license-compliance-brightgreen.svg)
28
28
  ![PyPI Version](https://img.shields.io/pypi/v/valx)
29
29
 
30
- An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Also now includes personal information detection and removal. Now includes hate speech and offensive language detection using AI.
30
+ An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. It also now provides detection and removal of personal information. Now includes hate speech and offensive language detection using AI.
31
+
32
+ > [!IMPORTANT]
33
+ > Please downgrade to `numpy` version `1.26.4`. Our ValX **DecisionTreeClassifier** AI model, relies on lower versions of `numpy`, because it was trained on these versions.
34
+ > For more information see: https://techoverflow.net/2024/07/23/how-to-fix-numpy-dtype-size-changed-may-indicate-binary-incompatibility-expected-96-from-c-header-got-88-from-pyobject/
35
+
36
+ ## Changes in 0.2.2
37
+
38
+ We have refactored and changed the `detect_profanity` function:
39
+ - Removed unnecessary printing
40
+ - Now returns more information about each found profanity, including `Line`, `Column`, `Word`, and `Language`.
41
+
42
+ > [!NOTE]
43
+ > You can view [ValX's package documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more information on changes.
31
44
 
32
45
  ## Changes in 0.2.1
33
46
 
@@ -63,6 +76,38 @@ Please ensure that you have one of these Python versions installed before using
63
76
  - **Detect Hate Speech**: Detect hate speech or offensive speech in text, using AI.
64
77
  - **Remove Hate Speech**: Remove hate speech or offensive speech in text, using AI.
65
78
 
79
+ ### List of supported languages for profanity detection and removal
80
+ Below is a complete list of all the available supported languages for ValX's profanity detection and removal functions which are valid values for `language`:
81
+
82
+ - **All**
83
+ - Arabic
84
+ - Czech
85
+ - Danish
86
+ - German
87
+ - English
88
+ - Esperanto
89
+ - Persian
90
+ - Finnish
91
+ - Filipino
92
+ - French
93
+ - French (CA)
94
+ - Hindi
95
+ - Hungarian
96
+ - Italian
97
+ - Japanese
98
+ - Kabyle
99
+ - Korean
100
+ - Dutch
101
+ - Norwegian
102
+ - Polish
103
+ - Portuguese
104
+ - Russian
105
+ - Swedish
106
+ - Thai
107
+ - Klingon
108
+ - Turkish
109
+ - Chinese
110
+
66
111
  ## Usage
67
112
 
68
113
  ### Detect Profanity
@@ -71,7 +116,8 @@ Please ensure that you have one of these Python versions installed before using
71
116
  from valx import detect_profanity
72
117
 
73
118
  # Detect profanity
74
- num_profanities = detect_profanity(sample_text, language='English')
119
+ results = detect_profanity(sample_text, language='English')
120
+ print("Profanity Evaluation Results", results)
75
121
  ```
76
122
 
77
123
  ### Remove Profanity
@@ -110,7 +156,14 @@ from valx import detect_hate_speech
110
156
  outcome_of_detection = detect_hate_speech("You are stupid.")
111
157
  ```
112
158
 
113
- > See our [official documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more examples on how to use ValX.*
159
+ > [!IMPORTANT]
160
+ > The model's possible outputs are:
161
+ > - `['Hate Speech']`: The text was flagged and contained hate speech.
162
+ > - `['Offensive Speech']`: The text was flagged and contained offensive speech.
163
+ > - `['No Hate and Offensive Speech']`: The text was not flagged for any hate speech or offensive speech.
164
+
165
+ > [!NOTE]
166
+ > See our [official documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more examples on how to use **ValX**.
114
167
 
115
168
  ## Contributing
116
169
 
@@ -5,7 +5,20 @@
5
5
  ![License Compliance](https://img.shields.io/badge/license-compliance-brightgreen.svg)
6
6
  ![PyPI Version](https://img.shields.io/pypi/v/valx)
7
7
 
8
- An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Also now includes personal information detection and removal. Now includes hate speech and offensive language detection using AI.
8
+ An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. It also now provides detection and removal of personal information. Now includes hate speech and offensive language detection using AI.
9
+
10
+ > [!IMPORTANT]
11
+ > Please downgrade to `numpy` version `1.26.4`. Our ValX **DecisionTreeClassifier** AI model, relies on lower versions of `numpy`, because it was trained on these versions.
12
+ > For more information see: https://techoverflow.net/2024/07/23/how-to-fix-numpy-dtype-size-changed-may-indicate-binary-incompatibility-expected-96-from-c-header-got-88-from-pyobject/
13
+
14
+ ## Changes in 0.2.2
15
+
16
+ We have refactored and changed the `detect_profanity` function:
17
+ - Removed unnecessary printing
18
+ - Now returns more information about each found profanity, including `Line`, `Column`, `Word`, and `Language`.
19
+
20
+ > [!NOTE]
21
+ > You can view [ValX's package documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more information on changes.
9
22
 
10
23
  ## Changes in 0.2.1
11
24
 
@@ -41,6 +54,38 @@ Please ensure that you have one of these Python versions installed before using
41
54
  - **Detect Hate Speech**: Detect hate speech or offensive speech in text, using AI.
42
55
  - **Remove Hate Speech**: Remove hate speech or offensive speech in text, using AI.
43
56
 
57
+ ### List of supported languages for profanity detection and removal
58
+ Below is a complete list of all the available supported languages for ValX's profanity detection and removal functions which are valid values for `language`:
59
+
60
+ - **All**
61
+ - Arabic
62
+ - Czech
63
+ - Danish
64
+ - German
65
+ - English
66
+ - Esperanto
67
+ - Persian
68
+ - Finnish
69
+ - Filipino
70
+ - French
71
+ - French (CA)
72
+ - Hindi
73
+ - Hungarian
74
+ - Italian
75
+ - Japanese
76
+ - Kabyle
77
+ - Korean
78
+ - Dutch
79
+ - Norwegian
80
+ - Polish
81
+ - Portuguese
82
+ - Russian
83
+ - Swedish
84
+ - Thai
85
+ - Klingon
86
+ - Turkish
87
+ - Chinese
88
+
44
89
  ## Usage
45
90
 
46
91
  ### Detect Profanity
@@ -49,7 +94,8 @@ Please ensure that you have one of these Python versions installed before using
49
94
  from valx import detect_profanity
50
95
 
51
96
  # Detect profanity
52
- num_profanities = detect_profanity(sample_text, language='English')
97
+ results = detect_profanity(sample_text, language='English')
98
+ print("Profanity Evaluation Results", results)
53
99
  ```
54
100
 
55
101
  ### Remove Profanity
@@ -88,7 +134,14 @@ from valx import detect_hate_speech
88
134
  outcome_of_detection = detect_hate_speech("You are stupid.")
89
135
  ```
90
136
 
91
- > See our [official documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more examples on how to use ValX.*
137
+ > [!IMPORTANT]
138
+ > The model's possible outputs are:
139
+ > - `['Hate Speech']`: The text was flagged and contained hate speech.
140
+ > - `['Offensive Speech']`: The text was flagged and contained offensive speech.
141
+ > - `['No Hate and Offensive Speech']`: The text was not flagged for any hate speech or offensive speech.
142
+
143
+ > [!NOTE]
144
+ > See our [official documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more examples on how to use **ValX**.
92
145
 
93
146
  ## Contributing
94
147
 
@@ -5,7 +5,7 @@ with open("README.md", "r", encoding="utf-8") as fh:
5
5
 
6
6
  setup(
7
7
  name='valx',
8
- version='0.2.1',
8
+ version='0.2.2',
9
9
  author='Infinitode Pty Ltd',
10
10
  author_email='infinitode.ltd@gmail.com',
11
11
  description='An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.',
@@ -2754,11 +2754,14 @@ def detect_profanity(text_data, language='English'):
2754
2754
 
2755
2755
  Args:
2756
2756
  text_data (list): A list of strings representing the text data to analyze.
2757
- language (str): The language for which to detect profanity. Defaults to 'English'.
2758
- allowed_languages (list): A list of languages to allow for detection. If provided, only profanity words for these languages will be considered.
2759
-
2757
+ language (str): The language used to detect profanity. Defaults to 'English'. Available languages include: All, Arabic, Czech, Danish, German, English, Esperanto, Persian, Finnish, Filipino, French, French (CA), Hindi, Hungarian, Italian, Japanese, Kabyle, Korean, Dutch, Norwegian, Polish, Portuguese, Russian, Swedish, Thai, Klingon, Turkish, Chinese.
2760
2758
  Returns:
2761
- int: The total number of profanity detections.
2759
+ list: A list of dictionaries where each dictionary represents a detected instance of profanity.
2760
+ Each dictionary contains the following keys:
2761
+ - "Line" (int): The line number where the profanity was detected.
2762
+ - "Column" (int): The column number (position in the line) where the profanity starts.
2763
+ - "Word" (str): The detected profanity word.
2764
+ - "Language" (str): The language in which the profanity was detected.
2762
2765
  """
2763
2766
  profanity_keywords = load_profanity_words(language)
2764
2767
  detected = set() # Use a set to store unique occurrences
@@ -2774,12 +2777,19 @@ def detect_profanity(text_data, language='English'):
2774
2777
  for match in matches:
2775
2778
  detected.add((i + 1, match.start() + 1, language, profanity)) # Add to set
2776
2779
 
2780
+ detections = [] # Initialize detections as a list
2781
+
2777
2782
  if detected:
2778
- print(f"Profanity detected:")
2779
2783
  for line_num, col_num, lang, word in detected:
2780
- print(f"Language: {lang}, Line {line_num}, Column {col_num}: '{word}'")
2784
+ detection_info = {
2785
+ "Line": line_num,
2786
+ "Column": col_num,
2787
+ "Word": word,
2788
+ "Language": lang
2789
+ }
2790
+ detections.append(detection_info) # Append each detection to the list
2781
2791
 
2782
- return len(detected)
2792
+ return detections
2783
2793
 
2784
2794
 
2785
2795
  def remove_profanity(text_data, output_file=None, language='English'):
@@ -2927,9 +2937,6 @@ def detect_hate_speech(text):
2927
2937
  Returns:
2928
2938
  list of str: A list of strings representing the outcome of the detection.
2929
2939
  """
2930
- # if isinstance(text, str):
2931
- # text = [text]
2932
-
2933
2940
  text = cv.transform([text]).toarray()
2934
2941
  return model.predict((text))
2935
2942
 
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.1
2
2
  Name: valx
3
- Version: 0.2.1
3
+ Version: 0.2.2
4
4
  Summary: An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.
5
5
  Home-page: https://github.com/infinitode/valx
6
6
  Author: Infinitode Pty Ltd
@@ -27,7 +27,20 @@ Requires-Dist: scikit-learn==1.2.2
27
27
  ![License Compliance](https://img.shields.io/badge/license-compliance-brightgreen.svg)
28
28
  ![PyPI Version](https://img.shields.io/pypi/v/valx)
29
29
 
30
- An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Also now includes personal information detection and removal. Now includes hate speech and offensive language detection using AI.
30
+ An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. It also now provides detection and removal of personal information. Now includes hate speech and offensive language detection using AI.
31
+
32
+ > [!IMPORTANT]
33
+ > Please downgrade to `numpy` version `1.26.4`. Our ValX **DecisionTreeClassifier** AI model, relies on lower versions of `numpy`, because it was trained on these versions.
34
+ > For more information see: https://techoverflow.net/2024/07/23/how-to-fix-numpy-dtype-size-changed-may-indicate-binary-incompatibility-expected-96-from-c-header-got-88-from-pyobject/
35
+
36
+ ## Changes in 0.2.2
37
+
38
+ We have refactored and changed the `detect_profanity` function:
39
+ - Removed unnecessary printing
40
+ - Now returns more information about each found profanity, including `Line`, `Column`, `Word`, and `Language`.
41
+
42
+ > [!NOTE]
43
+ > You can view [ValX's package documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more information on changes.
31
44
 
32
45
  ## Changes in 0.2.1
33
46
 
@@ -63,6 +76,38 @@ Please ensure that you have one of these Python versions installed before using
63
76
  - **Detect Hate Speech**: Detect hate speech or offensive speech in text, using AI.
64
77
  - **Remove Hate Speech**: Remove hate speech or offensive speech in text, using AI.
65
78
 
79
+ ### List of supported languages for profanity detection and removal
80
+ Below is a complete list of all the available supported languages for ValX's profanity detection and removal functions which are valid values for `language`:
81
+
82
+ - **All**
83
+ - Arabic
84
+ - Czech
85
+ - Danish
86
+ - German
87
+ - English
88
+ - Esperanto
89
+ - Persian
90
+ - Finnish
91
+ - Filipino
92
+ - French
93
+ - French (CA)
94
+ - Hindi
95
+ - Hungarian
96
+ - Italian
97
+ - Japanese
98
+ - Kabyle
99
+ - Korean
100
+ - Dutch
101
+ - Norwegian
102
+ - Polish
103
+ - Portuguese
104
+ - Russian
105
+ - Swedish
106
+ - Thai
107
+ - Klingon
108
+ - Turkish
109
+ - Chinese
110
+
66
111
  ## Usage
67
112
 
68
113
  ### Detect Profanity
@@ -71,7 +116,8 @@ Please ensure that you have one of these Python versions installed before using
71
116
  from valx import detect_profanity
72
117
 
73
118
  # Detect profanity
74
- num_profanities = detect_profanity(sample_text, language='English')
119
+ results = detect_profanity(sample_text, language='English')
120
+ print("Profanity Evaluation Results", results)
75
121
  ```
76
122
 
77
123
  ### Remove Profanity
@@ -110,7 +156,14 @@ from valx import detect_hate_speech
110
156
  outcome_of_detection = detect_hate_speech("You are stupid.")
111
157
  ```
112
158
 
113
- > See our [official documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more examples on how to use ValX.*
159
+ > [!IMPORTANT]
160
+ > The model's possible outputs are:
161
+ > - `['Hate Speech']`: The text was flagged and contained hate speech.
162
+ > - `['Offensive Speech']`: The text was flagged and contained offensive speech.
163
+ > - `['No Hate and Offensive Speech']`: The text was not flagged for any hate speech or offensive speech.
164
+
165
+ > [!NOTE]
166
+ > See our [official documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more examples on how to use **ValX**.
114
167
 
115
168
  ## Contributing
116
169
 
File without changes
File without changes
File without changes
File without changes
File without changes