valx 0.2.3__tar.gz → 0.2.5__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,11 +1,11 @@
1
- Metadata-Version: 2.1
1
+ Metadata-Version: 2.4
2
2
  Name: valx
3
- Version: 0.2.3
3
+ Version: 0.2.5
4
4
  Summary: An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.
5
5
  Home-page: https://github.com/infinitode/valx
6
6
  Author: Infinitode Pty Ltd
7
7
  Author-email: infinitode.ltd@gmail.com
8
- Classifier: Development Status :: 3 - Alpha
8
+ Classifier: Development Status :: 5 - Production/Stable
9
9
  Classifier: Intended Audience :: Developers
10
10
  Classifier: License :: OSI Approved :: MIT License
11
11
  Classifier: Programming Language :: Python :: 3.6
@@ -18,7 +18,17 @@ Classifier: Programming Language :: Python :: 3.12
18
18
  Requires-Python: >=3.6
19
19
  Description-Content-Type: text/markdown
20
20
  License-File: LICENSE
21
- Requires-Dist: scikit-learn==1.2.2
21
+ Requires-Dist: scikit-learn
22
+ Dynamic: author
23
+ Dynamic: author-email
24
+ Dynamic: classifier
25
+ Dynamic: description
26
+ Dynamic: description-content-type
27
+ Dynamic: home-page
28
+ Dynamic: license-file
29
+ Dynamic: requires-dist
30
+ Dynamic: requires-python
31
+ Dynamic: summary
22
32
 
23
33
  # ValX
24
34
  ![Python Version](https://img.shields.io/badge/python-3.12-blue.svg)
@@ -30,9 +40,35 @@ Requires-Dist: scikit-learn==1.2.2
30
40
  An open-source Python library for data cleaning tasks. It includes functions for profanity detection, and removal, and detection and removal of personal information. Also includes hate speech and offensive language detection and removal, using AI.
31
41
 
32
42
  > [!IMPORTANT]
43
+ > If you are using `scikit-learn` versions older than version `1.3.0`, please also downgrade your version of `numpy` as stated below. Otherwise, you can continue to use your preferred version of `scikit-learn` without downgrading `numpy`.
44
+ >
33
45
  > Please downgrade to `numpy` version `1.26.4`. Our ValX **DecisionTreeClassifier** AI model, relies on lower versions of `numpy`, because it was trained on these versions.
34
46
  > For more information see: https://techoverflow.net/2024/07/23/how-to-fix-numpy-dtype-size-changed-may-indicate-binary-incompatibility-expected-96-from-c-header-got-88-from-pyobject/
35
47
 
48
+ > [!NOTE]
49
+ > ValX will automatically install a version of `scikit-learn` that is compatible with your device if you don't have one already.
50
+
51
+ ## Changes in 0.2.5
52
+
53
+ ValX v0.2.5 introduces enhanced flexibility for profanity filtering by adding support for custom profanity lists:
54
+
55
+ - **Custom Profanity Word Lists**: Users can now provide their own lists of profane words directly as Python lists to the `detect_profanity` and `remove_profanity` functions via the new `custom_words_list` parameter.
56
+ - **Standalone Custom Lists**: Utilize your custom profanity list exclusively by setting the `language` parameter to `None`. ValX will then only use the words provided in `custom_words_list`.
57
+ - **Combined Lists**: Use a custom list in conjunction with ValX's built-in language-specific wordlists. Simply provide both a `language` (e.g., "English") and your `custom_words_list`. ValX will use the combined set of words.
58
+ - **Loading Custom Lists from File**: A new helper function, `load_custom_profanity_from_file(filepath)`, allows you to easily load custom profanity words from a text file.
59
+ - **File Format**: The file should contain one profanity word per line.
60
+ - Lines starting with a hash symbol (`#`) are treated as comments and ignored.
61
+ - Empty lines or lines containing only whitespace are also ignored.
62
+ - **Updated Detection Reporting**: The `detect_profanity` function's output now specifies the source of detected profanity more clearly (e.g., "Custom", "Custom + English").
63
+
64
+ These features give users greater control over the profanity filtering process, allowing for more tailored and specific use cases.
65
+
66
+ ## Changes in 0.2.4
67
+
68
+ Fixed a major incompatibility issue with `scikit-learn` due to version changes in `scikit-learn v1.3.0` which causes compatibility issues with versions later than `1.2.2`. ValX can now be used with `scikit-learn` versions earlier and later than `1.3.0`!
69
+
70
+ We've also removed `scikit-learn==1.2.2` as a dependency, as most versions of `scikit-learn` will now work.
71
+
36
72
  ## Changes in 0.2.3
37
73
 
38
74
  We have introduced a new optional `info_type` parameter into our `detect_sensitive_information`, and `remove_sensitive_information` functions, to allow you to have fine-grained control over what sensitive information you want to detect or remove.
@@ -124,25 +160,129 @@ Below is a complete list of all the available supported languages for ValX's pro
124
160
 
125
161
  ## Usage
126
162
 
127
- ### Detect Profanity
163
+ ### Profanity Detection and Removal
164
+
165
+ ValX allows for flexible profanity filtering using built-in language lists, custom word lists (provided as Python lists or loaded from files), or a combination of both.
166
+
167
+ **1. Basic Profanity Detection (Built-in Language)**
128
168
 
129
169
  ```python
130
170
  from valx import detect_profanity
131
171
 
132
- # Detect profanity
172
+ sample_text = ["This is some fuck and porn text."]
173
+ # Detect profanity using the English list
133
174
  results = detect_profanity(sample_text, language='English')
134
- print("Profanity Evaluation Results", results)
175
+ # results will be:
176
+ # [
177
+ # {'Line': 1, 'Column': 14, 'Word': 'fuck', 'Language': 'English'},
178
+ # {'Line': 1, 'Column': 23, 'Word': 'porn', 'Language': 'English'}
179
+ # ]
180
+ print(results)
135
181
  ```
136
182
 
137
- ### Remove Profanity
183
+ **2. Profanity Detection with a Custom Word List (Python List)**
184
+
185
+ You can provide your own list of words to filter.
138
186
 
139
187
  ```python
140
- from valx import remove_profanity
188
+ from valx import detect_profanity
141
189
 
142
- # Remove profanity
143
- removed = remove_profanity(sample_text, "text_cleaned.txt", language="English")
190
+ sample_text = ["This contains custombadword1 and also asshole from English list."]
191
+ my_custom_words = ["custombadword1", "anothercustom"]
192
+
193
+ # Option A: Custom list ONLY (language=None)
194
+ results_custom_only = detect_profanity(sample_text, language=None, custom_words_list=my_custom_words)
195
+ # results_custom_only will detect "custombadword1" with Language: "Custom"
196
+ # [{'Line': 1, 'Column': 15, 'Word': 'custombadword1', 'Language': 'Custom'}]
197
+ print(results_custom_only)
198
+
199
+ # Option B: Custom list COMBINED with a built-in language
200
+ results_custom_plus_english = detect_profanity(sample_text, language="English", custom_words_list=my_custom_words)
201
+ # results_custom_plus_english will detect "custombadword1" and "asshole"
202
+ # Language will be "Custom + English"
203
+ # [
204
+ # {'Line': 1, 'Column': 15, 'Word': 'custombadword1', 'Language': 'Custom + English'},
205
+ # {'Line': 1, 'Column': 43, 'Word': 'asshole', 'Language': 'Custom + English'}
206
+ # ]
207
+ print(results_custom_plus_english)
144
208
  ```
145
209
 
210
+ **3. Loading Custom Profanity Words from a File**
211
+
212
+ ValX provides a helper function to load words from a text file (one word per line, '#' for comments).
213
+
214
+ ```python
215
+ from valx import detect_profanity, load_custom_profanity_from_file
216
+
217
+ # Assume 'my_profanity_file.txt' contains:
218
+ # customfileword1
219
+ # # this is a comment
220
+ # customfileword2
221
+
222
+ custom_words_from_file = load_custom_profanity_from_file("my_profanity_file.txt")
223
+ # custom_words_from_file will be: ['customfileword1', 'customfileword2']
224
+
225
+ sample_text_for_file = ["Text with customfileword1 and built-in shit."]
226
+
227
+ # Use file-loaded list with English built-in list
228
+ results_file_plus_english = detect_profanity(
229
+ sample_text_for_file,
230
+ language="English",
231
+ custom_words_list=custom_words_from_file
232
+ )
233
+ # Detects "customfileword1" and "shit", Language: "Custom + English"
234
+ print(results_file_plus_english)
235
+
236
+ # Use file-loaded list ONLY
237
+ results_file_only = detect_profanity(
238
+ sample_text_for_file,
239
+ language=None, # Important: set language to None
240
+ custom_words_list=custom_words_from_file
241
+ )
242
+ # Detects only "customfileword1", Language: "Custom"
243
+ print(results_file_only)
244
+ ```
245
+
246
+ **Output Format for `detect_profanity`**
247
+
248
+ The `detect_profanity` function returns a list of dictionaries. Each dictionary includes:
249
+ - `"Line"`: The line number (1-indexed).
250
+ - `"Column"`: The column number (1-indexed) where the profanity starts.
251
+ - `"Word"`: The detected profanity word.
252
+ - `"Language"`: Indicates the source of the word list:
253
+ - `<LanguageName>` (e.g., "English"): If only a built-in language list was used.
254
+ - `"Custom"`: If `language=None` and only a `custom_words_list` was used.
255
+ - `"Custom + <LanguageName>"` (e.g., "Custom + English"): If both a built-in list and `custom_words_list` were used.
256
+ - `"Custom + All"`: If `language='All'` and `custom_words_list` were used.
257
+
258
+
259
+ **4. Removing Profanity**
260
+
261
+ `remove_profanity` works similarly, accepting `language` and `custom_words_list` parameters.
262
+
263
+ ```python
264
+ from valx import remove_profanity, load_custom_profanity_from_file
265
+
266
+ sample_text = ["This is fuck, custombadword1, and text with customfileword1."]
267
+ my_custom_words = ["custombadword1"]
268
+ custom_words_from_file = load_custom_profanity_from_file("my_profanity_file.txt") # Assuming it contains 'customfileword1'
269
+
270
+ # Remove profanity using English built-in + my_custom_words + custom_words_from_file
271
+ all_custom_words = list(set(my_custom_words + custom_words_from_file)) # Combine and unique
272
+
273
+ cleaned_text = remove_profanity(
274
+ sample_text,
275
+ output_file="cleaned_output.txt", # Optional: saves to file
276
+ language="English",
277
+ custom_words_list=all_custom_words
278
+ )
279
+ # cleaned_text will have "fuck", "custombadword1", and "customfileword1" replaced with "bad word".
280
+ # e.g., ["This is bad word, bad word, and text with bad word."]
281
+ print(cleaned_text)
282
+ ```
283
+
284
+ The `load_profanity_words` function (used internally) also accepts `language` and `custom_words_list` if you need direct access to the word lists.
285
+
146
286
  ### Detect Sensitive Information
147
287
 
148
288
  ```python
valx-0.2.5/README.md ADDED
@@ -0,0 +1,314 @@
1
+ # ValX
2
+ ![Python Version](https://img.shields.io/badge/python-3.12-blue.svg)
3
+ [![Code Size](https://img.shields.io/github/languages/code-size/infinitode/valx)](https://github.com/infinitode/valx)
4
+ ![Downloads](https://pepy.tech/badge/valx)
5
+ ![License Compliance](https://img.shields.io/badge/license-compliance-brightgreen.svg)
6
+ ![PyPI Version](https://img.shields.io/pypi/v/valx)
7
+
8
+ An open-source Python library for data cleaning tasks. It includes functions for profanity detection, and removal, and detection and removal of personal information. Also includes hate speech and offensive language detection and removal, using AI.
9
+
10
+ > [!IMPORTANT]
11
+ > If you are using `scikit-learn` versions older than version `1.3.0`, please also downgrade your version of `numpy` as stated below. Otherwise, you can continue to use your preferred version of `scikit-learn` without downgrading `numpy`.
12
+ >
13
+ > Please downgrade to `numpy` version `1.26.4`. Our ValX **DecisionTreeClassifier** AI model, relies on lower versions of `numpy`, because it was trained on these versions.
14
+ > For more information see: https://techoverflow.net/2024/07/23/how-to-fix-numpy-dtype-size-changed-may-indicate-binary-incompatibility-expected-96-from-c-header-got-88-from-pyobject/
15
+
16
+ > [!NOTE]
17
+ > ValX will automatically install a version of `scikit-learn` that is compatible with your device if you don't have one already.
18
+
19
+ ## Changes in 0.2.5
20
+
21
+ ValX v0.2.5 introduces enhanced flexibility for profanity filtering by adding support for custom profanity lists:
22
+
23
+ - **Custom Profanity Word Lists**: Users can now provide their own lists of profane words directly as Python lists to the `detect_profanity` and `remove_profanity` functions via the new `custom_words_list` parameter.
24
+ - **Standalone Custom Lists**: Utilize your custom profanity list exclusively by setting the `language` parameter to `None`. ValX will then only use the words provided in `custom_words_list`.
25
+ - **Combined Lists**: Use a custom list in conjunction with ValX's built-in language-specific wordlists. Simply provide both a `language` (e.g., "English") and your `custom_words_list`. ValX will use the combined set of words.
26
+ - **Loading Custom Lists from File**: A new helper function, `load_custom_profanity_from_file(filepath)`, allows you to easily load custom profanity words from a text file.
27
+ - **File Format**: The file should contain one profanity word per line.
28
+ - Lines starting with a hash symbol (`#`) are treated as comments and ignored.
29
+ - Empty lines or lines containing only whitespace are also ignored.
30
+ - **Updated Detection Reporting**: The `detect_profanity` function's output now specifies the source of detected profanity more clearly (e.g., "Custom", "Custom + English").
31
+
32
+ These features give users greater control over the profanity filtering process, allowing for more tailored and specific use cases.
33
+
34
+ ## Changes in 0.2.4
35
+
36
+ Fixed a major incompatibility issue with `scikit-learn` due to version changes in `scikit-learn v1.3.0` which causes compatibility issues with versions later than `1.2.2`. ValX can now be used with `scikit-learn` versions earlier and later than `1.3.0`!
37
+
38
+ We've also removed `scikit-learn==1.2.2` as a dependency, as most versions of `scikit-learn` will now work.
39
+
40
+ ## Changes in 0.2.3
41
+
42
+ We have introduced a new optional `info_type` parameter into our `detect_sensitive_information`, and `remove_sensitive_information` functions, to allow you to have fine-grained control over what sensitive information you want to detect or remove.
43
+
44
+ Also introduced more detection patterns for other types of sensitive information, including:
45
+ - `"iban"`: International Bank Account Number.
46
+ - `"mrn"`: Medical Record Number (may not work correctly, depending on provider and country).
47
+ - `"icd10"`: International Classification of Diseases, Tenth Revision.
48
+ - `"geo_coords"`: Geo-coordinates (latitude and longitude in decimal degrees format).
49
+ - `"username"`: Username handles (@username).
50
+ - `"file_path"`: File paths (general patterns for both Windows and Unix paths).
51
+ - `"bitcoin_wallet"`: Cryptocurrency wallet address.
52
+ - `"ethereum_wallet"`: Cryptocurrency wallet addresses.
53
+
54
+ ## Changes in 0.2.2
55
+
56
+ We have refactored and changed the `detect_profanity` function:
57
+ - Removed unnecessary printing
58
+ - Now returns more information about each found profanity, including `Line`, `Column`, `Word`, and `Language`.
59
+
60
+ > [!NOTE]
61
+ > You can view [ValX's package documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more information on changes.
62
+
63
+ ## Changes in 0.2.1
64
+
65
+ Using the AI models in ValX, you can now automatically remove hate speech, or offensive speech from your text data, without needing to run detection and write your own custom implementation method.
66
+
67
+ ## Installation
68
+
69
+ You can install ValX using pip:
70
+
71
+ ```bash
72
+ pip install valx
73
+ ```
74
+
75
+ ## Supported Python Versions
76
+
77
+ ValX supports the following Python versions:
78
+
79
+ - Python 3.6
80
+ - Python 3.7
81
+ - Python 3.8
82
+ - Python 3.9
83
+ - Python 3.10
84
+ - Python 3.11/Later (Preferred)
85
+
86
+ Please ensure that you have one of these Python versions installed before using ValX. ValX may not work as expected on lower versions of Python than the supported.
87
+
88
+ ## Features
89
+
90
+ - **Profanity Detection**: Detect profane and NSFW words or terms.
91
+ - **Remove Profanity**: Remove profane and NSFW words or terms.
92
+ - **Detect Sensitive Information**: Detect sensitive information in text data.
93
+ - **Remove Sensitive Information**: Remove sensitive information from text data.
94
+ - **Detect Hate Speech**: Detect hate speech or offensive speech in text, using AI.
95
+ - **Remove Hate Speech**: Remove hate speech or offensive speech in text, using AI.
96
+
97
+ ### List of supported languages for profanity detection and removal
98
+ Below is a complete list of all the available supported languages for ValX's profanity detection and removal functions which are valid values for `language`:
99
+
100
+ - **All**
101
+ - Arabic
102
+ - Czech
103
+ - Danish
104
+ - German
105
+ - English
106
+ - Esperanto
107
+ - Persian
108
+ - Finnish
109
+ - Filipino
110
+ - French
111
+ - French (CA)
112
+ - Hindi
113
+ - Hungarian
114
+ - Italian
115
+ - Japanese
116
+ - Kabyle
117
+ - Korean
118
+ - Dutch
119
+ - Norwegian
120
+ - Polish
121
+ - Portuguese
122
+ - Russian
123
+ - Swedish
124
+ - Thai
125
+ - Klingon
126
+ - Turkish
127
+ - Chinese
128
+
129
+ ## Usage
130
+
131
+ ### Profanity Detection and Removal
132
+
133
+ ValX allows for flexible profanity filtering using built-in language lists, custom word lists (provided as Python lists or loaded from files), or a combination of both.
134
+
135
+ **1. Basic Profanity Detection (Built-in Language)**
136
+
137
+ ```python
138
+ from valx import detect_profanity
139
+
140
+ sample_text = ["This is some fuck and porn text."]
141
+ # Detect profanity using the English list
142
+ results = detect_profanity(sample_text, language='English')
143
+ # results will be:
144
+ # [
145
+ # {'Line': 1, 'Column': 14, 'Word': 'fuck', 'Language': 'English'},
146
+ # {'Line': 1, 'Column': 23, 'Word': 'porn', 'Language': 'English'}
147
+ # ]
148
+ print(results)
149
+ ```
150
+
151
+ **2. Profanity Detection with a Custom Word List (Python List)**
152
+
153
+ You can provide your own list of words to filter.
154
+
155
+ ```python
156
+ from valx import detect_profanity
157
+
158
+ sample_text = ["This contains custombadword1 and also asshole from English list."]
159
+ my_custom_words = ["custombadword1", "anothercustom"]
160
+
161
+ # Option A: Custom list ONLY (language=None)
162
+ results_custom_only = detect_profanity(sample_text, language=None, custom_words_list=my_custom_words)
163
+ # results_custom_only will detect "custombadword1" with Language: "Custom"
164
+ # [{'Line': 1, 'Column': 15, 'Word': 'custombadword1', 'Language': 'Custom'}]
165
+ print(results_custom_only)
166
+
167
+ # Option B: Custom list COMBINED with a built-in language
168
+ results_custom_plus_english = detect_profanity(sample_text, language="English", custom_words_list=my_custom_words)
169
+ # results_custom_plus_english will detect "custombadword1" and "asshole"
170
+ # Language will be "Custom + English"
171
+ # [
172
+ # {'Line': 1, 'Column': 15, 'Word': 'custombadword1', 'Language': 'Custom + English'},
173
+ # {'Line': 1, 'Column': 43, 'Word': 'asshole', 'Language': 'Custom + English'}
174
+ # ]
175
+ print(results_custom_plus_english)
176
+ ```
177
+
178
+ **3. Loading Custom Profanity Words from a File**
179
+
180
+ ValX provides a helper function to load words from a text file (one word per line, '#' for comments).
181
+
182
+ ```python
183
+ from valx import detect_profanity, load_custom_profanity_from_file
184
+
185
+ # Assume 'my_profanity_file.txt' contains:
186
+ # customfileword1
187
+ # # this is a comment
188
+ # customfileword2
189
+
190
+ custom_words_from_file = load_custom_profanity_from_file("my_profanity_file.txt")
191
+ # custom_words_from_file will be: ['customfileword1', 'customfileword2']
192
+
193
+ sample_text_for_file = ["Text with customfileword1 and built-in shit."]
194
+
195
+ # Use file-loaded list with English built-in list
196
+ results_file_plus_english = detect_profanity(
197
+ sample_text_for_file,
198
+ language="English",
199
+ custom_words_list=custom_words_from_file
200
+ )
201
+ # Detects "customfileword1" and "shit", Language: "Custom + English"
202
+ print(results_file_plus_english)
203
+
204
+ # Use file-loaded list ONLY
205
+ results_file_only = detect_profanity(
206
+ sample_text_for_file,
207
+ language=None, # Important: set language to None
208
+ custom_words_list=custom_words_from_file
209
+ )
210
+ # Detects only "customfileword1", Language: "Custom"
211
+ print(results_file_only)
212
+ ```
213
+
214
+ **Output Format for `detect_profanity`**
215
+
216
+ The `detect_profanity` function returns a list of dictionaries. Each dictionary includes:
217
+ - `"Line"`: The line number (1-indexed).
218
+ - `"Column"`: The column number (1-indexed) where the profanity starts.
219
+ - `"Word"`: The detected profanity word.
220
+ - `"Language"`: Indicates the source of the word list:
221
+ - `<LanguageName>` (e.g., "English"): If only a built-in language list was used.
222
+ - `"Custom"`: If `language=None` and only a `custom_words_list` was used.
223
+ - `"Custom + <LanguageName>"` (e.g., "Custom + English"): If both a built-in list and `custom_words_list` were used.
224
+ - `"Custom + All"`: If `language='All'` and `custom_words_list` were used.
225
+
226
+
227
+ **4. Removing Profanity**
228
+
229
+ `remove_profanity` works similarly, accepting `language` and `custom_words_list` parameters.
230
+
231
+ ```python
232
+ from valx import remove_profanity, load_custom_profanity_from_file
233
+
234
+ sample_text = ["This is fuck, custombadword1, and text with customfileword1."]
235
+ my_custom_words = ["custombadword1"]
236
+ custom_words_from_file = load_custom_profanity_from_file("my_profanity_file.txt") # Assuming it contains 'customfileword1'
237
+
238
+ # Remove profanity using English built-in + my_custom_words + custom_words_from_file
239
+ all_custom_words = list(set(my_custom_words + custom_words_from_file)) # Combine and unique
240
+
241
+ cleaned_text = remove_profanity(
242
+ sample_text,
243
+ output_file="cleaned_output.txt", # Optional: saves to file
244
+ language="English",
245
+ custom_words_list=all_custom_words
246
+ )
247
+ # cleaned_text will have "fuck", "custombadword1", and "customfileword1" replaced with "bad word".
248
+ # e.g., ["This is bad word, bad word, and text with bad word."]
249
+ print(cleaned_text)
250
+ ```
251
+
252
+ The `load_profanity_words` function (used internally) also accepts `language` and `custom_words_list` if you need direct access to the word lists.
253
+
254
+ ### Detect Sensitive Information
255
+
256
+ ```python
257
+ from valx import detect_sensitive_information
258
+
259
+ # Detect sensitive information
260
+ detected_sensitive_info = detect_sensitive_information(sample_text)
261
+ ```
262
+
263
+ > [!NOTE]
264
+ > We have updated this function, and it now includes an optional argument for `info_type`, which can be used to detect only specific types of sensitive information. It was also added to `remove_sensitive_information`.
265
+
266
+ ### Remove Sensitive Information
267
+
268
+ ```python
269
+ from valx import remove_sensitive_information
270
+
271
+ # Remove sensitive information
272
+ cleaned_text = remove_sensitive_information(sample_text2)
273
+ ```
274
+
275
+ ### Detect Hate Speech And Offensive Language
276
+
277
+ ```python
278
+ from valx import detect_hate_speech
279
+
280
+ # Detect hate speech or offensive language
281
+ outcome_of_detection = detect_hate_speech("You are stupid.")
282
+ ```
283
+
284
+ > [!IMPORTANT]
285
+ > The model's possible outputs are:
286
+ > - `['Hate Speech']`: The text was flagged and contained hate speech.
287
+ > - `['Offensive Speech']`: The text was flagged and contained offensive speech.
288
+ > - `['No Hate and Offensive Speech']`: The text was not flagged for any hate speech or offensive speech.
289
+
290
+ > [!NOTE]
291
+ > See our [official documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more examples on how to use **ValX**.
292
+
293
+ ## Contributing
294
+
295
+ Contributions are welcome! If you encounter any issues, have suggestions, or want to contribute to ValX, please open an issue or submit a pull request on [GitHub](https://github.com/infinitode/valx).
296
+
297
+ ## License
298
+
299
+ ValX is released under the terms of the **MIT License (Modified)**. Please see the [LICENSE](https://github.com/infinitode/valx/blob/main/LICENSE) file for the full text.
300
+
301
+ ### Derived licenses
302
+ ---
303
+ ValX uses data from this GitHub repository:
304
+ https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words/
305
+ © 2012-2020 Shutterstock, Inc.
306
+
307
+ Creative Commons Attribution 4.0 International License:
308
+ https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words/blob/master/LICENSE
309
+
310
+ ---
311
+
312
+ **Modified License Clause**
313
+
314
+ The modified license clause grants users the permission to make derivative works based on the ValX software. However, it requires any substantial changes to the software to be clearly distinguished from the original work and distributed under a different name.
@@ -5,7 +5,7 @@ with open("README.md", "r", encoding="utf-8") as fh:
5
5
 
6
6
  setup(
7
7
  name='valx',
8
- version='0.2.3',
8
+ version='0.2.5',
9
9
  author='Infinitode Pty Ltd',
10
10
  author_email='infinitode.ltd@gmail.com',
11
11
  description='An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.',
@@ -15,10 +15,10 @@ setup(
15
15
  packages=find_packages(),
16
16
  package_data={'valx': ['models/*']},
17
17
  install_requires=[
18
- 'scikit-learn==1.2.2' # for the AI to function properly
18
+ 'scikit-learn'
19
19
  ],
20
20
  classifiers=[
21
- 'Development Status :: 3 - Alpha',
21
+ 'Development Status :: 5 - Production/Stable',
22
22
  'Intended Audience :: Developers',
23
23
  'License :: OSI Approved :: MIT License',
24
24
  'Programming Language :: Python :: 3.6',
@@ -3,6 +3,7 @@ import re
3
3
 
4
4
  # New version, that includes AI detection
5
5
  import pickle
6
+ from sklearn import __version__ as sklearnversion
6
7
  from sklearn.tree import DecisionTreeClassifier # Import the DecisionTreeClassifier class
7
8
  from sklearn.feature_extraction.text import CountVectorizer # Import the CountVectorizer class
8
9
 
@@ -15,15 +16,20 @@ Creative Commons Attribution 4.0 International License:
15
16
  https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words/blob/master/LICENSE
16
17
  '''
17
18
 
18
- def load_profanity_words(language='English'):
19
+ def load_profanity_words(language='English', custom_words_list: list[str] = None):
19
20
  """
20
- Load profanity words for the specified language.
21
+ Load profanity words for the specified language, optionally including a custom list.
21
22
 
22
23
  Args:
23
- language (str): The language for which to load profanity words. Defaults to 'English'.
24
+ language (str, optional): The language for which to load profanity words.
25
+ Defaults to 'English'. Can be None to use only custom_words_list.
26
+ custom_words_list (list[str], optional): A list of custom profanity words. Defaults to None.
24
27
 
25
28
  Returns:
26
- list: A list of profanity words for the specified language, or all languages if 'All' is specified.
29
+ list: A list of profanity words.
30
+
31
+ Raises:
32
+ ValueError: If language is None and custom_words_list is None or empty.
27
33
  """
28
34
  # Profanity words as a string
29
35
  PROFANITY_WORDS = """
@@ -2739,35 +2745,102 @@ def load_profanity_words(language='English'):
2739
2745
  if current_language:
2740
2746
  profanity_lists[current_language] = current_words
2741
2747
 
2742
- if language == 'All':
2743
- all_profanity_words = []
2748
+ loaded_words = []
2749
+ if language is None:
2750
+ if not custom_words_list:
2751
+ raise ValueError("If language is None, custom_words_list must be provided and non-empty.")
2752
+ loaded_words = list(custom_words_list) # Use a copy
2753
+ elif language == 'All':
2744
2754
  for words in profanity_lists.values():
2745
- all_profanity_words.extend(words)
2746
- return all_profanity_words
2755
+ loaded_words.extend(words)
2756
+ if custom_words_list:
2757
+ loaded_words.extend(w for w in custom_words_list if w not in loaded_words)
2747
2758
  else:
2748
- return profanity_lists.get(language, [])
2759
+ loaded_words = list(profanity_lists.get(language, [])) # Use a copy
2760
+ if custom_words_list:
2761
+ loaded_words.extend(w for w in custom_words_list if w not in loaded_words)
2762
+
2763
+ return list(set(loaded_words)) # Return unique words
2764
+
2765
+
2766
+ def load_custom_profanity_from_file(filepath: str) -> list[str]:
2767
+ """
2768
+ Loads a custom list of profanity words from a file.
2749
2769
 
2770
+ The file should contain one word per line. Lines starting with '#' will be ignored as comments.
2771
+ Empty lines or lines with only whitespace will also be ignored.
2772
+
2773
+ Args:
2774
+ filepath (str): The path to the file containing custom profanity words.
2775
+
2776
+ Returns:
2777
+ list[str]: A list of words loaded from the file.
2750
2778
 
2751
- def detect_profanity(text_data, language='English'):
2779
+ Raises:
2780
+ FileNotFoundError: If the specified filepath does not exist.
2781
+ IOError: For other file reading issues.
2782
+ """
2783
+ custom_words = []
2784
+ try:
2785
+ with open(filepath, 'r', encoding='utf-8') as f:
2786
+ for line in f:
2787
+ word = line.strip()
2788
+ if word and not word.startswith('#'):
2789
+ custom_words.append(word)
2790
+ except FileNotFoundError:
2791
+ # Re-raise FileNotFoundError to be explicit, or handle as per specific project policy
2792
+ # For example, could log a warning and return empty list:
2793
+ # print(f"Warning: File not found at {filepath}. Returning empty custom word list.")
2794
+ # return []
2795
+ raise
2796
+ except IOError as e:
2797
+ # Handle other IO errors, perhaps log them or raise a more generic error
2798
+ # print(f"Warning: Error reading file {filepath}: {e}. Returning empty custom word list.")
2799
+ # return []
2800
+ raise IOError(f"Error reading custom profanity file {filepath}: {e}")
2801
+ return custom_words
2802
+
2803
+
2804
+ def detect_profanity(text_data, language='English', custom_words_list: list[str] = None):
2752
2805
  """
2753
2806
  Detect profanity in text data using regex.
2754
2807
 
2755
2808
  Args:
2756
2809
  text_data (list): A list of strings representing the text data to analyze.
2757
- language (str): The language used to detect profanity. Defaults to 'English'. Available languages include: All, Arabic, Czech, Danish, German, English, Esperanto, Persian, Finnish, Filipino, French, French (CA), Hindi, Hungarian, Italian, Japanese, Kabyle, Korean, Dutch, Norwegian, Polish, Portuguese, Russian, Swedish, Thai, Klingon, Turkish, Chinese.
2810
+ language (str, optional): The language used to detect profanity. Defaults to 'English'.
2811
+ Can be None if custom_words_list is provided.
2812
+ Available languages include: All, Arabic, Czech, Danish, German,
2813
+ English, Esperanto, Persian, Finnish, Filipino, French,
2814
+ French (CA), Hindi, Hungarian, Italian, Japanese, Kabyle,
2815
+ Korean, Dutch, Norwegian, Polish, Portuguese, Russian,
2816
+ Swedish, Thai, Klingon, Turkish, Chinese.
2817
+ custom_words_list (list[str], optional): A list of custom profanity words. Defaults to None.
2758
2818
  Returns:
2759
2819
  list: A list of dictionaries where each dictionary represents a detected instance of profanity.
2760
2820
  Each dictionary contains the following keys:
2761
2821
  - "Line" (int): The line number where the profanity was detected.
2762
2822
  - "Column" (int): The column number (position in the line) where the profanity starts.
2763
2823
  - "Word" (str): The detected profanity word.
2764
- - "Language" (str): The language in which the profanity was detected.
2824
+ - "Language" (str): The language string indicating the source of the profanity list.
2765
2825
  """
2766
- profanity_keywords = load_profanity_words(language)
2826
+ profanity_keywords = load_profanity_words(language, custom_words_list)
2767
2827
  detected = set() # Use a set to store unique occurrences
2768
2828
 
2829
+ display_language = "Custom"
2830
+ if language:
2831
+ if custom_words_list:
2832
+ display_language = f"Custom + {language}"
2833
+ else:
2834
+ display_language = language
2835
+ elif not custom_words_list:
2836
+ # This case should ideally be prevented by load_profanity_words raising an error,
2837
+ # but as a fallback, if somehow load_profanity_words returns empty and language is None
2838
+ # and no custom_words_list, we avoid error here.
2839
+ return []
2840
+
2841
+
2769
2842
  for i, line in enumerate(text_data):
2770
- # Skip lines with language markers
2843
+ # Skip lines with language markers (though less relevant if custom lists are primary)
2771
2844
  if line.startswith("$") and line.endswith("$"):
2772
2845
  continue
2773
2846
 
@@ -2775,36 +2848,39 @@ def detect_profanity(text_data, language='English'):
2775
2848
  for profanity in profanity_keywords:
2776
2849
  matches = re.finditer(r'\b{}\b'.format(re.escape(profanity)), line, flags=re.IGNORECASE)
2777
2850
  for match in matches:
2778
- detected.add((i + 1, match.start() + 1, language, profanity)) # Add to set
2851
+ # Store with display_language, not the input `language` parameter
2852
+ detected.add((i + 1, match.start() + 1, display_language, profanity))
2779
2853
 
2780
2854
  detections = [] # Initialize detections as a list
2781
2855
 
2782
2856
  if detected:
2783
- for line_num, col_num, lang, word in detected:
2857
+ for line_num, col_num, lang_display_name, word in detected:
2784
2858
  detection_info = {
2785
2859
  "Line": line_num,
2786
2860
  "Column": col_num,
2787
2861
  "Word": word,
2788
- "Language": lang
2862
+ "Language": lang_display_name
2789
2863
  }
2790
- detections.append(detection_info) # Append each detection to the list
2864
+ detections.append(detection_info)
2791
2865
 
2792
2866
  return detections
2793
2867
 
2794
2868
 
2795
- def remove_profanity(text_data, output_file=None, language='English'):
2869
+ def remove_profanity(text_data, output_file=None, language='English', custom_words_list: list[str] = None):
2796
2870
  """
2797
2871
  Remove profanity from text data.
2798
2872
 
2799
2873
  Args:
2800
2874
  text_data (list): A list of strings representing the text data to clean.
2801
- output_file (str): The file path to write the cleaned data. If None, cleaned data is not written to a file.
2802
- language (str): The language for which to remove profanity. Defaults to 'English'.
2875
+ output_file (str, optional): The file path to write the cleaned data. If None, cleaned data is not written to a file.
2876
+ language (str, optional): The language for which to remove profanity. Defaults to 'English'.
2877
+ Can be None if custom_words_list is provided.
2878
+ custom_words_list (list[str], optional): A list of custom profanity words. Defaults to None.
2803
2879
 
2804
2880
  Returns:
2805
2881
  list: A list of strings representing the cleaned text data.
2806
2882
  """
2807
- profanity_keywords = load_profanity_words(language)
2883
+ profanity_keywords = load_profanity_words(language, custom_words_list)
2808
2884
  cleaned_data = []
2809
2885
 
2810
2886
  for line in text_data:
@@ -2902,8 +2978,18 @@ def remove_sensitive_information(text_data, output_file=None, info_type=["email"
2902
2978
 
2903
2979
  # New version
2904
2980
  MODEL_DIR = os.path.join(os.path.dirname(__file__), 'models')
2905
- DECISION_TREE_MODEL_PATH = os.path.join(MODEL_DIR, 'decision_tree_model.sav')
2906
2981
  COUNT_VECTORIZER_PATH = os.path.join(MODEL_DIR, 'count_vectorizer.sav')
2982
+ DEFAULT_MODEL_FILENAME = "decision_tree_model.sav"
2983
+ UPGRADED_MODEL_FILENAME = "classifier_upgraded.pkl"
2984
+
2985
+ # Check scikit-learn version
2986
+ SKLEARN_VERSION = tuple(map(int, sklearnversion.split(".")))
2987
+
2988
+ # Set the model path based on the version
2989
+ if SKLEARN_VERSION >= (1, 3, 0):
2990
+ DECISION_TREE_MODEL_PATH = os.path.join(MODEL_DIR, UPGRADED_MODEL_FILENAME)
2991
+ else:
2992
+ DECISION_TREE_MODEL_PATH = os.path.join(MODEL_DIR, DEFAULT_MODEL_FILENAME)
2907
2993
 
2908
2994
  # Load the saved models
2909
2995
  model = pickle.load(open(DECISION_TREE_MODEL_PATH, 'rb'))
@@ -1,11 +1,11 @@
1
- Metadata-Version: 2.1
1
+ Metadata-Version: 2.4
2
2
  Name: valx
3
- Version: 0.2.3
3
+ Version: 0.2.5
4
4
  Summary: An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.
5
5
  Home-page: https://github.com/infinitode/valx
6
6
  Author: Infinitode Pty Ltd
7
7
  Author-email: infinitode.ltd@gmail.com
8
- Classifier: Development Status :: 3 - Alpha
8
+ Classifier: Development Status :: 5 - Production/Stable
9
9
  Classifier: Intended Audience :: Developers
10
10
  Classifier: License :: OSI Approved :: MIT License
11
11
  Classifier: Programming Language :: Python :: 3.6
@@ -18,7 +18,17 @@ Classifier: Programming Language :: Python :: 3.12
18
18
  Requires-Python: >=3.6
19
19
  Description-Content-Type: text/markdown
20
20
  License-File: LICENSE
21
- Requires-Dist: scikit-learn==1.2.2
21
+ Requires-Dist: scikit-learn
22
+ Dynamic: author
23
+ Dynamic: author-email
24
+ Dynamic: classifier
25
+ Dynamic: description
26
+ Dynamic: description-content-type
27
+ Dynamic: home-page
28
+ Dynamic: license-file
29
+ Dynamic: requires-dist
30
+ Dynamic: requires-python
31
+ Dynamic: summary
22
32
 
23
33
  # ValX
24
34
  ![Python Version](https://img.shields.io/badge/python-3.12-blue.svg)
@@ -30,9 +40,35 @@ Requires-Dist: scikit-learn==1.2.2
30
40
  An open-source Python library for data cleaning tasks. It includes functions for profanity detection, and removal, and detection and removal of personal information. Also includes hate speech and offensive language detection and removal, using AI.
31
41
 
32
42
  > [!IMPORTANT]
43
+ > If you are using `scikit-learn` versions older than version `1.3.0`, please also downgrade your version of `numpy` as stated below. Otherwise, you can continue to use your preferred version of `scikit-learn` without downgrading `numpy`.
44
+ >
33
45
  > Please downgrade to `numpy` version `1.26.4`. Our ValX **DecisionTreeClassifier** AI model, relies on lower versions of `numpy`, because it was trained on these versions.
34
46
  > For more information see: https://techoverflow.net/2024/07/23/how-to-fix-numpy-dtype-size-changed-may-indicate-binary-incompatibility-expected-96-from-c-header-got-88-from-pyobject/
35
47
 
48
+ > [!NOTE]
49
+ > ValX will automatically install a version of `scikit-learn` that is compatible with your device if you don't have one already.
50
+
51
+ ## Changes in 0.2.5
52
+
53
+ ValX v0.2.5 introduces enhanced flexibility for profanity filtering by adding support for custom profanity lists:
54
+
55
+ - **Custom Profanity Word Lists**: Users can now provide their own lists of profane words directly as Python lists to the `detect_profanity` and `remove_profanity` functions via the new `custom_words_list` parameter.
56
+ - **Standalone Custom Lists**: Utilize your custom profanity list exclusively by setting the `language` parameter to `None`. ValX will then only use the words provided in `custom_words_list`.
57
+ - **Combined Lists**: Use a custom list in conjunction with ValX's built-in language-specific wordlists. Simply provide both a `language` (e.g., "English") and your `custom_words_list`. ValX will use the combined set of words.
58
+ - **Loading Custom Lists from File**: A new helper function, `load_custom_profanity_from_file(filepath)`, allows you to easily load custom profanity words from a text file.
59
+ - **File Format**: The file should contain one profanity word per line.
60
+ - Lines starting with a hash symbol (`#`) are treated as comments and ignored.
61
+ - Empty lines or lines containing only whitespace are also ignored.
62
+ - **Updated Detection Reporting**: The `detect_profanity` function's output now specifies the source of detected profanity more clearly (e.g., "Custom", "Custom + English").
63
+
64
+ These features give users greater control over the profanity filtering process, allowing for more tailored and specific use cases.
65
+
66
+ ## Changes in 0.2.4
67
+
68
+ Fixed a major incompatibility issue with `scikit-learn` due to version changes in `scikit-learn v1.3.0` which causes compatibility issues with versions later than `1.2.2`. ValX can now be used with `scikit-learn` versions earlier and later than `1.3.0`!
69
+
70
+ We've also removed `scikit-learn==1.2.2` as a dependency, as most versions of `scikit-learn` will now work.
71
+
36
72
  ## Changes in 0.2.3
37
73
 
38
74
  We have introduced a new optional `info_type` parameter into our `detect_sensitive_information`, and `remove_sensitive_information` functions, to allow you to have fine-grained control over what sensitive information you want to detect or remove.
@@ -124,25 +160,129 @@ Below is a complete list of all the available supported languages for ValX's pro
124
160
 
125
161
  ## Usage
126
162
 
127
- ### Detect Profanity
163
+ ### Profanity Detection and Removal
164
+
165
+ ValX allows for flexible profanity filtering using built-in language lists, custom word lists (provided as Python lists or loaded from files), or a combination of both.
166
+
167
+ **1. Basic Profanity Detection (Built-in Language)**
128
168
 
129
169
  ```python
130
170
  from valx import detect_profanity
131
171
 
132
- # Detect profanity
172
+ sample_text = ["This is some fuck and porn text."]
173
+ # Detect profanity using the English list
133
174
  results = detect_profanity(sample_text, language='English')
134
- print("Profanity Evaluation Results", results)
175
+ # results will be:
176
+ # [
177
+ # {'Line': 1, 'Column': 14, 'Word': 'fuck', 'Language': 'English'},
178
+ # {'Line': 1, 'Column': 23, 'Word': 'porn', 'Language': 'English'}
179
+ # ]
180
+ print(results)
135
181
  ```
136
182
 
137
- ### Remove Profanity
183
+ **2. Profanity Detection with a Custom Word List (Python List)**
184
+
185
+ You can provide your own list of words to filter.
138
186
 
139
187
  ```python
140
- from valx import remove_profanity
188
+ from valx import detect_profanity
141
189
 
142
- # Remove profanity
143
- removed = remove_profanity(sample_text, "text_cleaned.txt", language="English")
190
+ sample_text = ["This contains custombadword1 and also asshole from English list."]
191
+ my_custom_words = ["custombadword1", "anothercustom"]
192
+
193
+ # Option A: Custom list ONLY (language=None)
194
+ results_custom_only = detect_profanity(sample_text, language=None, custom_words_list=my_custom_words)
195
+ # results_custom_only will detect "custombadword1" with Language: "Custom"
196
+ # [{'Line': 1, 'Column': 15, 'Word': 'custombadword1', 'Language': 'Custom'}]
197
+ print(results_custom_only)
198
+
199
+ # Option B: Custom list COMBINED with a built-in language
200
+ results_custom_plus_english = detect_profanity(sample_text, language="English", custom_words_list=my_custom_words)
201
+ # results_custom_plus_english will detect "custombadword1" and "asshole"
202
+ # Language will be "Custom + English"
203
+ # [
204
+ # {'Line': 1, 'Column': 15, 'Word': 'custombadword1', 'Language': 'Custom + English'},
205
+ # {'Line': 1, 'Column': 43, 'Word': 'asshole', 'Language': 'Custom + English'}
206
+ # ]
207
+ print(results_custom_plus_english)
144
208
  ```
145
209
 
210
+ **3. Loading Custom Profanity Words from a File**
211
+
212
+ ValX provides a helper function to load words from a text file (one word per line, '#' for comments).
213
+
214
+ ```python
215
+ from valx import detect_profanity, load_custom_profanity_from_file
216
+
217
+ # Assume 'my_profanity_file.txt' contains:
218
+ # customfileword1
219
+ # # this is a comment
220
+ # customfileword2
221
+
222
+ custom_words_from_file = load_custom_profanity_from_file("my_profanity_file.txt")
223
+ # custom_words_from_file will be: ['customfileword1', 'customfileword2']
224
+
225
+ sample_text_for_file = ["Text with customfileword1 and built-in shit."]
226
+
227
+ # Use file-loaded list with English built-in list
228
+ results_file_plus_english = detect_profanity(
229
+ sample_text_for_file,
230
+ language="English",
231
+ custom_words_list=custom_words_from_file
232
+ )
233
+ # Detects "customfileword1" and "shit", Language: "Custom + English"
234
+ print(results_file_plus_english)
235
+
236
+ # Use file-loaded list ONLY
237
+ results_file_only = detect_profanity(
238
+ sample_text_for_file,
239
+ language=None, # Important: set language to None
240
+ custom_words_list=custom_words_from_file
241
+ )
242
+ # Detects only "customfileword1", Language: "Custom"
243
+ print(results_file_only)
244
+ ```
245
+
246
+ **Output Format for `detect_profanity`**
247
+
248
+ The `detect_profanity` function returns a list of dictionaries. Each dictionary includes:
249
+ - `"Line"`: The line number (1-indexed).
250
+ - `"Column"`: The column number (1-indexed) where the profanity starts.
251
+ - `"Word"`: The detected profanity word.
252
+ - `"Language"`: Indicates the source of the word list:
253
+ - `<LanguageName>` (e.g., "English"): If only a built-in language list was used.
254
+ - `"Custom"`: If `language=None` and only a `custom_words_list` was used.
255
+ - `"Custom + <LanguageName>"` (e.g., "Custom + English"): If both a built-in list and `custom_words_list` were used.
256
+ - `"Custom + All"`: If `language='All'` and `custom_words_list` were used.
257
+
258
+
259
+ **4. Removing Profanity**
260
+
261
+ `remove_profanity` works similarly, accepting `language` and `custom_words_list` parameters.
262
+
263
+ ```python
264
+ from valx import remove_profanity, load_custom_profanity_from_file
265
+
266
+ sample_text = ["This is fuck, custombadword1, and text with customfileword1."]
267
+ my_custom_words = ["custombadword1"]
268
+ custom_words_from_file = load_custom_profanity_from_file("my_profanity_file.txt") # Assuming it contains 'customfileword1'
269
+
270
+ # Remove profanity using English built-in + my_custom_words + custom_words_from_file
271
+ all_custom_words = list(set(my_custom_words + custom_words_from_file)) # Combine and unique
272
+
273
+ cleaned_text = remove_profanity(
274
+ sample_text,
275
+ output_file="cleaned_output.txt", # Optional: saves to file
276
+ language="English",
277
+ custom_words_list=all_custom_words
278
+ )
279
+ # cleaned_text will have "fuck", "custombadword1", and "customfileword1" replaced with "bad word".
280
+ # e.g., ["This is bad word, bad word, and text with bad word."]
281
+ print(cleaned_text)
282
+ ```
283
+
284
+ The `load_profanity_words` function (used internally) also accepts `language` and `custom_words_list` if you need direct access to the word lists.
285
+
146
286
  ### Detect Sensitive Information
147
287
 
148
288
  ```python
@@ -7,5 +7,6 @@ valx.egg-info/SOURCES.txt
7
7
  valx.egg-info/dependency_links.txt
8
8
  valx.egg-info/requires.txt
9
9
  valx.egg-info/top_level.txt
10
+ valx/models/classifier_upgraded.pkl
10
11
  valx/models/count_vectorizer.sav
11
12
  valx/models/decision_tree_model.sav
@@ -0,0 +1 @@
1
+ scikit-learn
valx-0.2.3/README.md DELETED
@@ -1,184 +0,0 @@
1
- # ValX
2
- ![Python Version](https://img.shields.io/badge/python-3.12-blue.svg)
3
- [![Code Size](https://img.shields.io/github/languages/code-size/infinitode/valx)](https://github.com/infinitode/valx)
4
- ![Downloads](https://pepy.tech/badge/valx)
5
- ![License Compliance](https://img.shields.io/badge/license-compliance-brightgreen.svg)
6
- ![PyPI Version](https://img.shields.io/pypi/v/valx)
7
-
8
- An open-source Python library for data cleaning tasks. It includes functions for profanity detection, and removal, and detection and removal of personal information. Also includes hate speech and offensive language detection and removal, using AI.
9
-
10
- > [!IMPORTANT]
11
- > Please downgrade to `numpy` version `1.26.4`. Our ValX **DecisionTreeClassifier** AI model, relies on lower versions of `numpy`, because it was trained on these versions.
12
- > For more information see: https://techoverflow.net/2024/07/23/how-to-fix-numpy-dtype-size-changed-may-indicate-binary-incompatibility-expected-96-from-c-header-got-88-from-pyobject/
13
-
14
- ## Changes in 0.2.3
15
-
16
- We have introduced a new optional `info_type` parameter into our `detect_sensitive_information`, and `remove_sensitive_information` functions, to allow you to have fine-grained control over what sensitive information you want to detect or remove.
17
-
18
- Also introduced more detection patterns for other types of sensitive information, including:
19
- - `"iban"`: International Bank Account Number.
20
- - `"mrn"`: Medical Record Number (may not work correctly, depending on provider and country).
21
- - `"icd10"`: International Classification of Diseases, Tenth Revision.
22
- - `"geo_coords"`: Geo-coordinates (latitude and longitude in decimal degrees format).
23
- - `"username"`: Username handles (@username).
24
- - `"file_path"`: File paths (general patterns for both Windows and Unix paths).
25
- - `"bitcoin_wallet"`: Cryptocurrency wallet address.
26
- - `"ethereum_wallet"`: Cryptocurrency wallet addresses.
27
-
28
- ## Changes in 0.2.2
29
-
30
- We have refactored and changed the `detect_profanity` function:
31
- - Removed unnecessary printing
32
- - Now returns more information about each found profanity, including `Line`, `Column`, `Word`, and `Language`.
33
-
34
- > [!NOTE]
35
- > You can view [ValX's package documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more information on changes.
36
-
37
- ## Changes in 0.2.1
38
-
39
- Using the AI models in ValX, you can now automatically remove hate speech, or offensive speech from your text data, without needing to run detection and write your own custom implementation method.
40
-
41
- ## Installation
42
-
43
- You can install ValX using pip:
44
-
45
- ```bash
46
- pip install valx
47
- ```
48
-
49
- ## Supported Python Versions
50
-
51
- ValX supports the following Python versions:
52
-
53
- - Python 3.6
54
- - Python 3.7
55
- - Python 3.8
56
- - Python 3.9
57
- - Python 3.10
58
- - Python 3.11/Later (Preferred)
59
-
60
- Please ensure that you have one of these Python versions installed before using ValX. ValX may not work as expected on lower versions of Python than the supported.
61
-
62
- ## Features
63
-
64
- - **Profanity Detection**: Detect profane and NSFW words or terms.
65
- - **Remove Profanity**: Remove profane and NSFW words or terms.
66
- - **Detect Sensitive Information**: Detect sensitive information in text data.
67
- - **Remove Sensitive Information**: Remove sensitive information from text data.
68
- - **Detect Hate Speech**: Detect hate speech or offensive speech in text, using AI.
69
- - **Remove Hate Speech**: Remove hate speech or offensive speech in text, using AI.
70
-
71
- ### List of supported languages for profanity detection and removal
72
- Below is a complete list of all the available supported languages for ValX's profanity detection and removal functions which are valid values for `language`:
73
-
74
- - **All**
75
- - Arabic
76
- - Czech
77
- - Danish
78
- - German
79
- - English
80
- - Esperanto
81
- - Persian
82
- - Finnish
83
- - Filipino
84
- - French
85
- - French (CA)
86
- - Hindi
87
- - Hungarian
88
- - Italian
89
- - Japanese
90
- - Kabyle
91
- - Korean
92
- - Dutch
93
- - Norwegian
94
- - Polish
95
- - Portuguese
96
- - Russian
97
- - Swedish
98
- - Thai
99
- - Klingon
100
- - Turkish
101
- - Chinese
102
-
103
- ## Usage
104
-
105
- ### Detect Profanity
106
-
107
- ```python
108
- from valx import detect_profanity
109
-
110
- # Detect profanity
111
- results = detect_profanity(sample_text, language='English')
112
- print("Profanity Evaluation Results", results)
113
- ```
114
-
115
- ### Remove Profanity
116
-
117
- ```python
118
- from valx import remove_profanity
119
-
120
- # Remove profanity
121
- removed = remove_profanity(sample_text, "text_cleaned.txt", language="English")
122
- ```
123
-
124
- ### Detect Sensitive Information
125
-
126
- ```python
127
- from valx import detect_sensitive_information
128
-
129
- # Detect sensitive information
130
- detected_sensitive_info = detect_sensitive_information(sample_text)
131
- ```
132
-
133
- > [!NOTE]
134
- > We have updated this function, and it now includes an optional argument for `info_type`, which can be used to detect only specific types of sensitive information. It was also added to `remove_sensitive_information`.
135
-
136
- ### Remove Sensitive Information
137
-
138
- ```python
139
- from valx import remove_sensitive_information
140
-
141
- # Remove sensitive information
142
- cleaned_text = remove_sensitive_information(sample_text2)
143
- ```
144
-
145
- ### Detect Hate Speech And Offensive Language
146
-
147
- ```python
148
- from valx import detect_hate_speech
149
-
150
- # Detect hate speech or offensive language
151
- outcome_of_detection = detect_hate_speech("You are stupid.")
152
- ```
153
-
154
- > [!IMPORTANT]
155
- > The model's possible outputs are:
156
- > - `['Hate Speech']`: The text was flagged and contained hate speech.
157
- > - `['Offensive Speech']`: The text was flagged and contained offensive speech.
158
- > - `['No Hate and Offensive Speech']`: The text was not flagged for any hate speech or offensive speech.
159
-
160
- > [!NOTE]
161
- > See our [official documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more examples on how to use **ValX**.
162
-
163
- ## Contributing
164
-
165
- Contributions are welcome! If you encounter any issues, have suggestions, or want to contribute to ValX, please open an issue or submit a pull request on [GitHub](https://github.com/infinitode/valx).
166
-
167
- ## License
168
-
169
- ValX is released under the terms of the **MIT License (Modified)**. Please see the [LICENSE](https://github.com/infinitode/valx/blob/main/LICENSE) file for the full text.
170
-
171
- ### Derived licenses
172
- ---
173
- ValX uses data from this GitHub repository:
174
- https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words/
175
- © 2012-2020 Shutterstock, Inc.
176
-
177
- Creative Commons Attribution 4.0 International License:
178
- https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words/blob/master/LICENSE
179
-
180
- ---
181
-
182
- **Modified License Clause**
183
-
184
- The modified license clause grants users the permission to make derivative works based on the ValX software. However, it requires any substantial changes to the software to be clearly distinguished from the original work and distributed under a different name.
@@ -1 +0,0 @@
1
- scikit-learn==1.2.2
File without changes
File without changes
File without changes