valx 0.2.1__tar.gz → 0.2.3__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {valx-0.2.1/valx.egg-info → valx-0.2.3}/PKG-INFO +74 -4
- {valx-0.2.1 → valx-0.2.3}/README.md +73 -3
- {valx-0.2.1 → valx-0.2.3}/setup.py +1 -1
- {valx-0.2.1 → valx-0.2.3}/valx/__init__.py +54 -65
- {valx-0.2.1 → valx-0.2.3/valx.egg-info}/PKG-INFO +74 -4
- {valx-0.2.1 → valx-0.2.3}/LICENSE +0 -0
- {valx-0.2.1 → valx-0.2.3}/setup.cfg +0 -0
- {valx-0.2.1 → valx-0.2.3}/valx/models/count_vectorizer.sav +0 -0
- {valx-0.2.1 → valx-0.2.3}/valx/models/decision_tree_model.sav +0 -0
- {valx-0.2.1 → valx-0.2.3}/valx.egg-info/SOURCES.txt +0 -0
- {valx-0.2.1 → valx-0.2.3}/valx.egg-info/dependency_links.txt +0 -0
- {valx-0.2.1 → valx-0.2.3}/valx.egg-info/requires.txt +0 -0
- {valx-0.2.1 → valx-0.2.3}/valx.egg-info/top_level.txt +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.1
|
|
2
2
|
Name: valx
|
|
3
|
-
Version: 0.2.
|
|
3
|
+
Version: 0.2.3
|
|
4
4
|
Summary: An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.
|
|
5
5
|
Home-page: https://github.com/infinitode/valx
|
|
6
6
|
Author: Infinitode Pty Ltd
|
|
@@ -27,7 +27,34 @@ Requires-Dist: scikit-learn==1.2.2
|
|
|
27
27
|

|
|
28
28
|

|
|
29
29
|
|
|
30
|
-
An open-source Python library for data cleaning tasks.
|
|
30
|
+
An open-source Python library for data cleaning tasks. It includes functions for profanity detection, and removal, and detection and removal of personal information. Also includes hate speech and offensive language detection and removal, using AI.
|
|
31
|
+
|
|
32
|
+
> [!IMPORTANT]
|
|
33
|
+
> Please downgrade to `numpy` version `1.26.4`. Our ValX **DecisionTreeClassifier** AI model, relies on lower versions of `numpy`, because it was trained on these versions.
|
|
34
|
+
> For more information see: https://techoverflow.net/2024/07/23/how-to-fix-numpy-dtype-size-changed-may-indicate-binary-incompatibility-expected-96-from-c-header-got-88-from-pyobject/
|
|
35
|
+
|
|
36
|
+
## Changes in 0.2.3
|
|
37
|
+
|
|
38
|
+
We have introduced a new optional `info_type` parameter into our `detect_sensitive_information`, and `remove_sensitive_information` functions, to allow you to have fine-grained control over what sensitive information you want to detect or remove.
|
|
39
|
+
|
|
40
|
+
Also introduced more detection patterns for other types of sensitive information, including:
|
|
41
|
+
- `"iban"`: International Bank Account Number.
|
|
42
|
+
- `"mrn"`: Medical Record Number (may not work correctly, depending on provider and country).
|
|
43
|
+
- `"icd10"`: International Classification of Diseases, Tenth Revision.
|
|
44
|
+
- `"geo_coords"`: Geo-coordinates (latitude and longitude in decimal degrees format).
|
|
45
|
+
- `"username"`: Username handles (@username).
|
|
46
|
+
- `"file_path"`: File paths (general patterns for both Windows and Unix paths).
|
|
47
|
+
- `"bitcoin_wallet"`: Cryptocurrency wallet address.
|
|
48
|
+
- `"ethereum_wallet"`: Cryptocurrency wallet addresses.
|
|
49
|
+
|
|
50
|
+
## Changes in 0.2.2
|
|
51
|
+
|
|
52
|
+
We have refactored and changed the `detect_profanity` function:
|
|
53
|
+
- Removed unnecessary printing
|
|
54
|
+
- Now returns more information about each found profanity, including `Line`, `Column`, `Word`, and `Language`.
|
|
55
|
+
|
|
56
|
+
> [!NOTE]
|
|
57
|
+
> You can view [ValX's package documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more information on changes.
|
|
31
58
|
|
|
32
59
|
## Changes in 0.2.1
|
|
33
60
|
|
|
@@ -63,6 +90,38 @@ Please ensure that you have one of these Python versions installed before using
|
|
|
63
90
|
- **Detect Hate Speech**: Detect hate speech or offensive speech in text, using AI.
|
|
64
91
|
- **Remove Hate Speech**: Remove hate speech or offensive speech in text, using AI.
|
|
65
92
|
|
|
93
|
+
### List of supported languages for profanity detection and removal
|
|
94
|
+
Below is a complete list of all the available supported languages for ValX's profanity detection and removal functions which are valid values for `language`:
|
|
95
|
+
|
|
96
|
+
- **All**
|
|
97
|
+
- Arabic
|
|
98
|
+
- Czech
|
|
99
|
+
- Danish
|
|
100
|
+
- German
|
|
101
|
+
- English
|
|
102
|
+
- Esperanto
|
|
103
|
+
- Persian
|
|
104
|
+
- Finnish
|
|
105
|
+
- Filipino
|
|
106
|
+
- French
|
|
107
|
+
- French (CA)
|
|
108
|
+
- Hindi
|
|
109
|
+
- Hungarian
|
|
110
|
+
- Italian
|
|
111
|
+
- Japanese
|
|
112
|
+
- Kabyle
|
|
113
|
+
- Korean
|
|
114
|
+
- Dutch
|
|
115
|
+
- Norwegian
|
|
116
|
+
- Polish
|
|
117
|
+
- Portuguese
|
|
118
|
+
- Russian
|
|
119
|
+
- Swedish
|
|
120
|
+
- Thai
|
|
121
|
+
- Klingon
|
|
122
|
+
- Turkish
|
|
123
|
+
- Chinese
|
|
124
|
+
|
|
66
125
|
## Usage
|
|
67
126
|
|
|
68
127
|
### Detect Profanity
|
|
@@ -71,7 +130,8 @@ Please ensure that you have one of these Python versions installed before using
|
|
|
71
130
|
from valx import detect_profanity
|
|
72
131
|
|
|
73
132
|
# Detect profanity
|
|
74
|
-
|
|
133
|
+
results = detect_profanity(sample_text, language='English')
|
|
134
|
+
print("Profanity Evaluation Results", results)
|
|
75
135
|
```
|
|
76
136
|
|
|
77
137
|
### Remove Profanity
|
|
@@ -92,6 +152,9 @@ from valx import detect_sensitive_information
|
|
|
92
152
|
detected_sensitive_info = detect_sensitive_information(sample_text)
|
|
93
153
|
```
|
|
94
154
|
|
|
155
|
+
> [!NOTE]
|
|
156
|
+
> We have updated this function, and it now includes an optional argument for `info_type`, which can be used to detect only specific types of sensitive information. It was also added to `remove_sensitive_information`.
|
|
157
|
+
|
|
95
158
|
### Remove Sensitive Information
|
|
96
159
|
|
|
97
160
|
```python
|
|
@@ -110,7 +173,14 @@ from valx import detect_hate_speech
|
|
|
110
173
|
outcome_of_detection = detect_hate_speech("You are stupid.")
|
|
111
174
|
```
|
|
112
175
|
|
|
113
|
-
>
|
|
176
|
+
> [!IMPORTANT]
|
|
177
|
+
> The model's possible outputs are:
|
|
178
|
+
> - `['Hate Speech']`: The text was flagged and contained hate speech.
|
|
179
|
+
> - `['Offensive Speech']`: The text was flagged and contained offensive speech.
|
|
180
|
+
> - `['No Hate and Offensive Speech']`: The text was not flagged for any hate speech or offensive speech.
|
|
181
|
+
|
|
182
|
+
> [!NOTE]
|
|
183
|
+
> See our [official documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more examples on how to use **ValX**.
|
|
114
184
|
|
|
115
185
|
## Contributing
|
|
116
186
|
|
|
@@ -5,7 +5,34 @@
|
|
|
5
5
|

|
|
6
6
|

|
|
7
7
|
|
|
8
|
-
An open-source Python library for data cleaning tasks.
|
|
8
|
+
An open-source Python library for data cleaning tasks. It includes functions for profanity detection, and removal, and detection and removal of personal information. Also includes hate speech and offensive language detection and removal, using AI.
|
|
9
|
+
|
|
10
|
+
> [!IMPORTANT]
|
|
11
|
+
> Please downgrade to `numpy` version `1.26.4`. Our ValX **DecisionTreeClassifier** AI model, relies on lower versions of `numpy`, because it was trained on these versions.
|
|
12
|
+
> For more information see: https://techoverflow.net/2024/07/23/how-to-fix-numpy-dtype-size-changed-may-indicate-binary-incompatibility-expected-96-from-c-header-got-88-from-pyobject/
|
|
13
|
+
|
|
14
|
+
## Changes in 0.2.3
|
|
15
|
+
|
|
16
|
+
We have introduced a new optional `info_type` parameter into our `detect_sensitive_information`, and `remove_sensitive_information` functions, to allow you to have fine-grained control over what sensitive information you want to detect or remove.
|
|
17
|
+
|
|
18
|
+
Also introduced more detection patterns for other types of sensitive information, including:
|
|
19
|
+
- `"iban"`: International Bank Account Number.
|
|
20
|
+
- `"mrn"`: Medical Record Number (may not work correctly, depending on provider and country).
|
|
21
|
+
- `"icd10"`: International Classification of Diseases, Tenth Revision.
|
|
22
|
+
- `"geo_coords"`: Geo-coordinates (latitude and longitude in decimal degrees format).
|
|
23
|
+
- `"username"`: Username handles (@username).
|
|
24
|
+
- `"file_path"`: File paths (general patterns for both Windows and Unix paths).
|
|
25
|
+
- `"bitcoin_wallet"`: Cryptocurrency wallet address.
|
|
26
|
+
- `"ethereum_wallet"`: Cryptocurrency wallet addresses.
|
|
27
|
+
|
|
28
|
+
## Changes in 0.2.2
|
|
29
|
+
|
|
30
|
+
We have refactored and changed the `detect_profanity` function:
|
|
31
|
+
- Removed unnecessary printing
|
|
32
|
+
- Now returns more information about each found profanity, including `Line`, `Column`, `Word`, and `Language`.
|
|
33
|
+
|
|
34
|
+
> [!NOTE]
|
|
35
|
+
> You can view [ValX's package documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more information on changes.
|
|
9
36
|
|
|
10
37
|
## Changes in 0.2.1
|
|
11
38
|
|
|
@@ -41,6 +68,38 @@ Please ensure that you have one of these Python versions installed before using
|
|
|
41
68
|
- **Detect Hate Speech**: Detect hate speech or offensive speech in text, using AI.
|
|
42
69
|
- **Remove Hate Speech**: Remove hate speech or offensive speech in text, using AI.
|
|
43
70
|
|
|
71
|
+
### List of supported languages for profanity detection and removal
|
|
72
|
+
Below is a complete list of all the available supported languages for ValX's profanity detection and removal functions which are valid values for `language`:
|
|
73
|
+
|
|
74
|
+
- **All**
|
|
75
|
+
- Arabic
|
|
76
|
+
- Czech
|
|
77
|
+
- Danish
|
|
78
|
+
- German
|
|
79
|
+
- English
|
|
80
|
+
- Esperanto
|
|
81
|
+
- Persian
|
|
82
|
+
- Finnish
|
|
83
|
+
- Filipino
|
|
84
|
+
- French
|
|
85
|
+
- French (CA)
|
|
86
|
+
- Hindi
|
|
87
|
+
- Hungarian
|
|
88
|
+
- Italian
|
|
89
|
+
- Japanese
|
|
90
|
+
- Kabyle
|
|
91
|
+
- Korean
|
|
92
|
+
- Dutch
|
|
93
|
+
- Norwegian
|
|
94
|
+
- Polish
|
|
95
|
+
- Portuguese
|
|
96
|
+
- Russian
|
|
97
|
+
- Swedish
|
|
98
|
+
- Thai
|
|
99
|
+
- Klingon
|
|
100
|
+
- Turkish
|
|
101
|
+
- Chinese
|
|
102
|
+
|
|
44
103
|
## Usage
|
|
45
104
|
|
|
46
105
|
### Detect Profanity
|
|
@@ -49,7 +108,8 @@ Please ensure that you have one of these Python versions installed before using
|
|
|
49
108
|
from valx import detect_profanity
|
|
50
109
|
|
|
51
110
|
# Detect profanity
|
|
52
|
-
|
|
111
|
+
results = detect_profanity(sample_text, language='English')
|
|
112
|
+
print("Profanity Evaluation Results", results)
|
|
53
113
|
```
|
|
54
114
|
|
|
55
115
|
### Remove Profanity
|
|
@@ -70,6 +130,9 @@ from valx import detect_sensitive_information
|
|
|
70
130
|
detected_sensitive_info = detect_sensitive_information(sample_text)
|
|
71
131
|
```
|
|
72
132
|
|
|
133
|
+
> [!NOTE]
|
|
134
|
+
> We have updated this function, and it now includes an optional argument for `info_type`, which can be used to detect only specific types of sensitive information. It was also added to `remove_sensitive_information`.
|
|
135
|
+
|
|
73
136
|
### Remove Sensitive Information
|
|
74
137
|
|
|
75
138
|
```python
|
|
@@ -88,7 +151,14 @@ from valx import detect_hate_speech
|
|
|
88
151
|
outcome_of_detection = detect_hate_speech("You are stupid.")
|
|
89
152
|
```
|
|
90
153
|
|
|
91
|
-
>
|
|
154
|
+
> [!IMPORTANT]
|
|
155
|
+
> The model's possible outputs are:
|
|
156
|
+
> - `['Hate Speech']`: The text was flagged and contained hate speech.
|
|
157
|
+
> - `['Offensive Speech']`: The text was flagged and contained offensive speech.
|
|
158
|
+
> - `['No Hate and Offensive Speech']`: The text was not flagged for any hate speech or offensive speech.
|
|
159
|
+
|
|
160
|
+
> [!NOTE]
|
|
161
|
+
> See our [official documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more examples on how to use **ValX**.
|
|
92
162
|
|
|
93
163
|
## Contributing
|
|
94
164
|
|
|
@@ -5,7 +5,7 @@ with open("README.md", "r", encoding="utf-8") as fh:
|
|
|
5
5
|
|
|
6
6
|
setup(
|
|
7
7
|
name='valx',
|
|
8
|
-
version='0.2.
|
|
8
|
+
version='0.2.3',
|
|
9
9
|
author='Infinitode Pty Ltd',
|
|
10
10
|
author_email='infinitode.ltd@gmail.com',
|
|
11
11
|
description='An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.',
|
|
@@ -2754,11 +2754,14 @@ def detect_profanity(text_data, language='English'):
|
|
|
2754
2754
|
|
|
2755
2755
|
Args:
|
|
2756
2756
|
text_data (list): A list of strings representing the text data to analyze.
|
|
2757
|
-
language (str): The language
|
|
2758
|
-
allowed_languages (list): A list of languages to allow for detection. If provided, only profanity words for these languages will be considered.
|
|
2759
|
-
|
|
2757
|
+
language (str): The language used to detect profanity. Defaults to 'English'. Available languages include: All, Arabic, Czech, Danish, German, English, Esperanto, Persian, Finnish, Filipino, French, French (CA), Hindi, Hungarian, Italian, Japanese, Kabyle, Korean, Dutch, Norwegian, Polish, Portuguese, Russian, Swedish, Thai, Klingon, Turkish, Chinese.
|
|
2760
2758
|
Returns:
|
|
2761
|
-
|
|
2759
|
+
list: A list of dictionaries where each dictionary represents a detected instance of profanity.
|
|
2760
|
+
Each dictionary contains the following keys:
|
|
2761
|
+
- "Line" (int): The line number where the profanity was detected.
|
|
2762
|
+
- "Column" (int): The column number (position in the line) where the profanity starts.
|
|
2763
|
+
- "Word" (str): The detected profanity word.
|
|
2764
|
+
- "Language" (str): The language in which the profanity was detected.
|
|
2762
2765
|
"""
|
|
2763
2766
|
profanity_keywords = load_profanity_words(language)
|
|
2764
2767
|
detected = set() # Use a set to store unique occurrences
|
|
@@ -2774,12 +2777,19 @@ def detect_profanity(text_data, language='English'):
|
|
|
2774
2777
|
for match in matches:
|
|
2775
2778
|
detected.add((i + 1, match.start() + 1, language, profanity)) # Add to set
|
|
2776
2779
|
|
|
2780
|
+
detections = [] # Initialize detections as a list
|
|
2781
|
+
|
|
2777
2782
|
if detected:
|
|
2778
|
-
print(f"Profanity detected:")
|
|
2779
2783
|
for line_num, col_num, lang, word in detected:
|
|
2780
|
-
|
|
2784
|
+
detection_info = {
|
|
2785
|
+
"Line": line_num,
|
|
2786
|
+
"Column": col_num,
|
|
2787
|
+
"Word": word,
|
|
2788
|
+
"Language": lang
|
|
2789
|
+
}
|
|
2790
|
+
detections.append(detection_info) # Append each detection to the list
|
|
2781
2791
|
|
|
2782
|
-
return
|
|
2792
|
+
return detections
|
|
2783
2793
|
|
|
2784
2794
|
|
|
2785
2795
|
def remove_profanity(text_data, output_file=None, language='English'):
|
|
@@ -2815,85 +2825,68 @@ def remove_profanity(text_data, output_file=None, language='English'):
|
|
|
2815
2825
|
|
|
2816
2826
|
return cleaned_data
|
|
2817
2827
|
|
|
2818
|
-
|
|
2819
|
-
def detect_sensitive_information(text_data):
|
|
2828
|
+
def detect_sensitive_information(text_data, info_type=["email", "phone", "credit_card", "ssn", "id", "address", "ip", "iban", "mrn", "icd10", "geo_coords", "username", "file_path", "bitcoin_wallet", "ethereum_wallet"]):
|
|
2820
2829
|
"""
|
|
2821
2830
|
Detect sensitive information patterns in the provided text data.
|
|
2822
2831
|
|
|
2823
2832
|
Args:
|
|
2824
2833
|
text_data (list of str): A list of strings representing the text data to be analyzed.
|
|
2834
|
+
info_type (str or list of str, optional): One or more types of sensitive info to detect. Available types are: "email", "phone", "credit_card", "ssn", "id", "address", "ip", "iban", "mrn", "icd10", "geo_coords", "username", "file_path", "bitcoin_wallet", "ethereum_wallet". Uses all info types by default.
|
|
2825
2835
|
|
|
2826
2836
|
Returns:
|
|
2827
2837
|
list of tuple: A list of tuples containing detected sensitive information, each tuple
|
|
2828
2838
|
representing (line number, column index, type, value).
|
|
2829
2839
|
"""
|
|
2840
|
+
if isinstance(info_type, str):
|
|
2841
|
+
info_type = [info_type] # Convert single string to list for consistent handling
|
|
2842
|
+
|
|
2843
|
+
# Dictionary to map info_type to regex patterns
|
|
2844
|
+
regex_patterns = {
|
|
2845
|
+
"email": r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b',
|
|
2846
|
+
"phone": r'\b(?:\d{3}[-.\s]??\d{3}[-.\s]??\d{4}|\(\d{3}\)\s*\d{3}[-.\s]??\d{4}|\d{3}[-.\s]??\d{4})\b',
|
|
2847
|
+
"credit_card": r'\b(?:\d[ -]*?){13,16}\b',
|
|
2848
|
+
"ssn": r'\b\d{3}[-]?\d{2}[-]?\d{4}\b',
|
|
2849
|
+
"id": r'\b[A-Za-z]{1,2}\d{6,9}\b',
|
|
2850
|
+
"address": r'\b\d+\s\w+\s\w+,\s\w+,\s\w+\b',
|
|
2851
|
+
"ip": r'\b(?:[0-9]{1,3}\.){3}[0-9]{1,3}\b',
|
|
2852
|
+
"iban": r'\b[A-Z]{2}[0-9]{2}[A-Z0-9]{1,30}\b',
|
|
2853
|
+
"mrn": r'\b\d{5,10}\b',
|
|
2854
|
+
"icd10": r'\b[A-TV-Z][0-9]{2}(?:\.[0-9A-TV-Z]{1,4})?\b',
|
|
2855
|
+
"geo_coords": r'\b[-+]?([1-8]?\d(\.\d+)?|90(\.0+)?),\s*[-+]?(180(\.0+)?|((1[0-7]\d)|([1-9]?\d))(\.\d+)?)\b',
|
|
2856
|
+
"username": r'@\w+',
|
|
2857
|
+
"file_path": r'([A-Za-z]:\\|/)?([-\w\s]+\\|/)*[-\w\s]+\.\w+',
|
|
2858
|
+
"bitcoin_wallet": r'\b[13][a-km-zA-HJ-NP-Z1-9]{25,34}\b',
|
|
2859
|
+
"ethereum_wallet": r'\b0x[a-fA-F0-9]{40}\b'
|
|
2860
|
+
}
|
|
2861
|
+
|
|
2830
2862
|
sensitive_info = []
|
|
2831
2863
|
|
|
2832
|
-
#
|
|
2833
|
-
email_regex = r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b'
|
|
2834
|
-
# Regular expression for detecting phone numbers
|
|
2835
|
-
phone_regex = r'\b(?:\d{3}[-.\s]??\d{3}[-.\s]??\d{4}|\(\d{3}\)\s*\d{3}[-.\s]??\d{4}|\d{3}[-.\s]??\d{4})\b'
|
|
2836
|
-
# Regular expression for detecting credit card numbers (Visa, Mastercard, American Express, Discover)
|
|
2837
|
-
credit_card_regex = r'\b(?:\d[ -]*?){13,16}\b'
|
|
2838
|
-
# Regular expression for detecting social security numbers (SSN)
|
|
2839
|
-
ssn_regex = r'\b\d{3}[-]?\d{2}[-]?\d{4}\b'
|
|
2840
|
-
# Regular expression for detecting ID numbers (e.g., driver's license, passport)
|
|
2841
|
-
id_number_regex = r'\b[A-Za-z]{1,2}\d{6,9}\b'
|
|
2842
|
-
# Regular expression for detecting addresses (simple example)
|
|
2843
|
-
address_regex = r'\b\d+\s\w+\s\w+,\s\w+,\s\w+\b'
|
|
2844
|
-
# Regular expression for detecting IP addresses
|
|
2845
|
-
ip_address_regex = r'\b(?:[0-9]{1,3}\.){3}[0-9]{1,3}\b'
|
|
2846
|
-
|
|
2864
|
+
# Loop through each line and check for sensitive info types based on specified `info_type`
|
|
2847
2865
|
for i, line in enumerate(text_data):
|
|
2848
|
-
|
|
2849
|
-
|
|
2850
|
-
|
|
2851
|
-
|
|
2852
|
-
|
|
2853
|
-
|
|
2854
|
-
|
|
2855
|
-
for phone in phones:
|
|
2856
|
-
sensitive_info.append((i + 1, line.find(phone), 'Phone', phone))
|
|
2857
|
-
|
|
2858
|
-
# Detect credit card numbers
|
|
2859
|
-
credit_cards = re.findall(credit_card_regex, line)
|
|
2860
|
-
for card in credit_cards:
|
|
2861
|
-
sensitive_info.append((i + 1, line.find(card), 'Credit Card', card))
|
|
2862
|
-
|
|
2863
|
-
# Detect social security numbers
|
|
2864
|
-
ssn_numbers = re.findall(ssn_regex, line)
|
|
2865
|
-
for ssn in ssn_numbers:
|
|
2866
|
-
sensitive_info.append((i + 1, line.find(ssn), 'SSN', ssn))
|
|
2867
|
-
|
|
2868
|
-
# Detect ID numbers
|
|
2869
|
-
id_numbers = re.findall(id_number_regex, line)
|
|
2870
|
-
for id_number in id_numbers:
|
|
2871
|
-
sensitive_info.append((i + 1, line.find(id_number), 'ID Number', id_number))
|
|
2872
|
-
|
|
2873
|
-
# Detect addresses
|
|
2874
|
-
addresses = re.findall(address_regex, line)
|
|
2875
|
-
for address in addresses:
|
|
2876
|
-
sensitive_info.append((i + 1, line.find(address), 'Address', address))
|
|
2877
|
-
|
|
2878
|
-
# Detect IP addresses
|
|
2879
|
-
ip_addresses = re.findall(ip_address_regex, line)
|
|
2880
|
-
for ip_address in ip_addresses:
|
|
2881
|
-
sensitive_info.append((i + 1, line.find(ip_address), 'IP Address', ip_address))
|
|
2882
|
-
|
|
2866
|
+
for type_ in info_type:
|
|
2867
|
+
pattern = regex_patterns.get(type_)
|
|
2868
|
+
if pattern:
|
|
2869
|
+
matches = re.finditer(pattern, line)
|
|
2870
|
+
for match in matches:
|
|
2871
|
+
sensitive_info.append((i + 1, match.start(), type_.capitalize(), match.group()))
|
|
2872
|
+
|
|
2883
2873
|
return sensitive_info
|
|
2884
2874
|
|
|
2885
|
-
def remove_sensitive_information(text_data, output_file=None):
|
|
2875
|
+
def remove_sensitive_information(text_data, output_file=None, info_type=["email", "phone", "credit_card", "ssn", "id", "address", "ip", "iban", "mrn", "icd10", "geo_coords", "username", "file_path", "bitcoin_wallet", "ethereum_wallet"]):
|
|
2886
2876
|
"""
|
|
2887
2877
|
Remove sensitive information patterns from the provided text data.
|
|
2888
2878
|
|
|
2889
2879
|
Args:
|
|
2890
2880
|
text_data (list of str): A list of strings representing the text data to be cleaned.
|
|
2891
2881
|
output_file (str, optional): Path to the output file where cleaned data will be saved.
|
|
2882
|
+
info_type (str or list of str, optional): One or more types of sensitive info to detect and remove. Available types are: "email", "phone", "credit_card", "ssn", "id", "address", "ip", "iban", "mrn", "icd10", "geo_coords", "username", "file_path", "bitcoin_wallet", "ethereum_wallet". Uses all info types by default.
|
|
2892
2883
|
|
|
2893
2884
|
Returns:
|
|
2894
2885
|
list of str: A list of strings representing the cleaned text data.
|
|
2895
2886
|
"""
|
|
2896
|
-
|
|
2887
|
+
# Detect sensitive information
|
|
2888
|
+
sensitive_info = detect_sensitive_information(text_data, info_type=info_type)
|
|
2889
|
+
|
|
2897
2890
|
cleaned_data = []
|
|
2898
2891
|
for line in text_data:
|
|
2899
2892
|
cleaned_line = line
|
|
@@ -2907,7 +2900,6 @@ def remove_sensitive_information(text_data, output_file=None):
|
|
|
2907
2900
|
file.write('\n'.join(cleaned_data))
|
|
2908
2901
|
return cleaned_data
|
|
2909
2902
|
|
|
2910
|
-
|
|
2911
2903
|
# New version
|
|
2912
2904
|
MODEL_DIR = os.path.join(os.path.dirname(__file__), 'models')
|
|
2913
2905
|
DECISION_TREE_MODEL_PATH = os.path.join(MODEL_DIR, 'decision_tree_model.sav')
|
|
@@ -2927,9 +2919,6 @@ def detect_hate_speech(text):
|
|
|
2927
2919
|
Returns:
|
|
2928
2920
|
list of str: A list of strings representing the outcome of the detection.
|
|
2929
2921
|
"""
|
|
2930
|
-
# if isinstance(text, str):
|
|
2931
|
-
# text = [text]
|
|
2932
|
-
|
|
2933
2922
|
text = cv.transform([text]).toarray()
|
|
2934
2923
|
return model.predict((text))
|
|
2935
2924
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.1
|
|
2
2
|
Name: valx
|
|
3
|
-
Version: 0.2.
|
|
3
|
+
Version: 0.2.3
|
|
4
4
|
Summary: An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.
|
|
5
5
|
Home-page: https://github.com/infinitode/valx
|
|
6
6
|
Author: Infinitode Pty Ltd
|
|
@@ -27,7 +27,34 @@ Requires-Dist: scikit-learn==1.2.2
|
|
|
27
27
|

|
|
28
28
|

|
|
29
29
|
|
|
30
|
-
An open-source Python library for data cleaning tasks.
|
|
30
|
+
An open-source Python library for data cleaning tasks. It includes functions for profanity detection, and removal, and detection and removal of personal information. Also includes hate speech and offensive language detection and removal, using AI.
|
|
31
|
+
|
|
32
|
+
> [!IMPORTANT]
|
|
33
|
+
> Please downgrade to `numpy` version `1.26.4`. Our ValX **DecisionTreeClassifier** AI model, relies on lower versions of `numpy`, because it was trained on these versions.
|
|
34
|
+
> For more information see: https://techoverflow.net/2024/07/23/how-to-fix-numpy-dtype-size-changed-may-indicate-binary-incompatibility-expected-96-from-c-header-got-88-from-pyobject/
|
|
35
|
+
|
|
36
|
+
## Changes in 0.2.3
|
|
37
|
+
|
|
38
|
+
We have introduced a new optional `info_type` parameter into our `detect_sensitive_information`, and `remove_sensitive_information` functions, to allow you to have fine-grained control over what sensitive information you want to detect or remove.
|
|
39
|
+
|
|
40
|
+
Also introduced more detection patterns for other types of sensitive information, including:
|
|
41
|
+
- `"iban"`: International Bank Account Number.
|
|
42
|
+
- `"mrn"`: Medical Record Number (may not work correctly, depending on provider and country).
|
|
43
|
+
- `"icd10"`: International Classification of Diseases, Tenth Revision.
|
|
44
|
+
- `"geo_coords"`: Geo-coordinates (latitude and longitude in decimal degrees format).
|
|
45
|
+
- `"username"`: Username handles (@username).
|
|
46
|
+
- `"file_path"`: File paths (general patterns for both Windows and Unix paths).
|
|
47
|
+
- `"bitcoin_wallet"`: Cryptocurrency wallet address.
|
|
48
|
+
- `"ethereum_wallet"`: Cryptocurrency wallet addresses.
|
|
49
|
+
|
|
50
|
+
## Changes in 0.2.2
|
|
51
|
+
|
|
52
|
+
We have refactored and changed the `detect_profanity` function:
|
|
53
|
+
- Removed unnecessary printing
|
|
54
|
+
- Now returns more information about each found profanity, including `Line`, `Column`, `Word`, and `Language`.
|
|
55
|
+
|
|
56
|
+
> [!NOTE]
|
|
57
|
+
> You can view [ValX's package documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more information on changes.
|
|
31
58
|
|
|
32
59
|
## Changes in 0.2.1
|
|
33
60
|
|
|
@@ -63,6 +90,38 @@ Please ensure that you have one of these Python versions installed before using
|
|
|
63
90
|
- **Detect Hate Speech**: Detect hate speech or offensive speech in text, using AI.
|
|
64
91
|
- **Remove Hate Speech**: Remove hate speech or offensive speech in text, using AI.
|
|
65
92
|
|
|
93
|
+
### List of supported languages for profanity detection and removal
|
|
94
|
+
Below is a complete list of all the available supported languages for ValX's profanity detection and removal functions which are valid values for `language`:
|
|
95
|
+
|
|
96
|
+
- **All**
|
|
97
|
+
- Arabic
|
|
98
|
+
- Czech
|
|
99
|
+
- Danish
|
|
100
|
+
- German
|
|
101
|
+
- English
|
|
102
|
+
- Esperanto
|
|
103
|
+
- Persian
|
|
104
|
+
- Finnish
|
|
105
|
+
- Filipino
|
|
106
|
+
- French
|
|
107
|
+
- French (CA)
|
|
108
|
+
- Hindi
|
|
109
|
+
- Hungarian
|
|
110
|
+
- Italian
|
|
111
|
+
- Japanese
|
|
112
|
+
- Kabyle
|
|
113
|
+
- Korean
|
|
114
|
+
- Dutch
|
|
115
|
+
- Norwegian
|
|
116
|
+
- Polish
|
|
117
|
+
- Portuguese
|
|
118
|
+
- Russian
|
|
119
|
+
- Swedish
|
|
120
|
+
- Thai
|
|
121
|
+
- Klingon
|
|
122
|
+
- Turkish
|
|
123
|
+
- Chinese
|
|
124
|
+
|
|
66
125
|
## Usage
|
|
67
126
|
|
|
68
127
|
### Detect Profanity
|
|
@@ -71,7 +130,8 @@ Please ensure that you have one of these Python versions installed before using
|
|
|
71
130
|
from valx import detect_profanity
|
|
72
131
|
|
|
73
132
|
# Detect profanity
|
|
74
|
-
|
|
133
|
+
results = detect_profanity(sample_text, language='English')
|
|
134
|
+
print("Profanity Evaluation Results", results)
|
|
75
135
|
```
|
|
76
136
|
|
|
77
137
|
### Remove Profanity
|
|
@@ -92,6 +152,9 @@ from valx import detect_sensitive_information
|
|
|
92
152
|
detected_sensitive_info = detect_sensitive_information(sample_text)
|
|
93
153
|
```
|
|
94
154
|
|
|
155
|
+
> [!NOTE]
|
|
156
|
+
> We have updated this function, and it now includes an optional argument for `info_type`, which can be used to detect only specific types of sensitive information. It was also added to `remove_sensitive_information`.
|
|
157
|
+
|
|
95
158
|
### Remove Sensitive Information
|
|
96
159
|
|
|
97
160
|
```python
|
|
@@ -110,7 +173,14 @@ from valx import detect_hate_speech
|
|
|
110
173
|
outcome_of_detection = detect_hate_speech("You are stupid.")
|
|
111
174
|
```
|
|
112
175
|
|
|
113
|
-
>
|
|
176
|
+
> [!IMPORTANT]
|
|
177
|
+
> The model's possible outputs are:
|
|
178
|
+
> - `['Hate Speech']`: The text was flagged and contained hate speech.
|
|
179
|
+
> - `['Offensive Speech']`: The text was flagged and contained offensive speech.
|
|
180
|
+
> - `['No Hate and Offensive Speech']`: The text was not flagged for any hate speech or offensive speech.
|
|
181
|
+
|
|
182
|
+
> [!NOTE]
|
|
183
|
+
> See our [official documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more examples on how to use **ValX**.
|
|
114
184
|
|
|
115
185
|
## Contributing
|
|
116
186
|
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|