valx 0.2.2__tar.gz → 0.2.3__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {valx-0.2.2/valx.egg-info → valx-0.2.3}/PKG-INFO +19 -2
- {valx-0.2.2 → valx-0.2.3}/README.md +18 -1
- {valx-0.2.2 → valx-0.2.3}/setup.py +1 -1
- {valx-0.2.2 → valx-0.2.3}/valx/__init__.py +37 -55
- {valx-0.2.2 → valx-0.2.3/valx.egg-info}/PKG-INFO +19 -2
- {valx-0.2.2 → valx-0.2.3}/LICENSE +0 -0
- {valx-0.2.2 → valx-0.2.3}/setup.cfg +0 -0
- {valx-0.2.2 → valx-0.2.3}/valx/models/count_vectorizer.sav +0 -0
- {valx-0.2.2 → valx-0.2.3}/valx/models/decision_tree_model.sav +0 -0
- {valx-0.2.2 → valx-0.2.3}/valx.egg-info/SOURCES.txt +0 -0
- {valx-0.2.2 → valx-0.2.3}/valx.egg-info/dependency_links.txt +0 -0
- {valx-0.2.2 → valx-0.2.3}/valx.egg-info/requires.txt +0 -0
- {valx-0.2.2 → valx-0.2.3}/valx.egg-info/top_level.txt +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.1
|
|
2
2
|
Name: valx
|
|
3
|
-
Version: 0.2.
|
|
3
|
+
Version: 0.2.3
|
|
4
4
|
Summary: An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.
|
|
5
5
|
Home-page: https://github.com/infinitode/valx
|
|
6
6
|
Author: Infinitode Pty Ltd
|
|
@@ -27,12 +27,26 @@ Requires-Dist: scikit-learn==1.2.2
|
|
|
27
27
|

|
|
28
28
|

|
|
29
29
|
|
|
30
|
-
An open-source Python library for data cleaning tasks.
|
|
30
|
+
An open-source Python library for data cleaning tasks. It includes functions for profanity detection, and removal, and detection and removal of personal information. Also includes hate speech and offensive language detection and removal, using AI.
|
|
31
31
|
|
|
32
32
|
> [!IMPORTANT]
|
|
33
33
|
> Please downgrade to `numpy` version `1.26.4`. Our ValX **DecisionTreeClassifier** AI model, relies on lower versions of `numpy`, because it was trained on these versions.
|
|
34
34
|
> For more information see: https://techoverflow.net/2024/07/23/how-to-fix-numpy-dtype-size-changed-may-indicate-binary-incompatibility-expected-96-from-c-header-got-88-from-pyobject/
|
|
35
35
|
|
|
36
|
+
## Changes in 0.2.3
|
|
37
|
+
|
|
38
|
+
We have introduced a new optional `info_type` parameter into our `detect_sensitive_information`, and `remove_sensitive_information` functions, to allow you to have fine-grained control over what sensitive information you want to detect or remove.
|
|
39
|
+
|
|
40
|
+
Also introduced more detection patterns for other types of sensitive information, including:
|
|
41
|
+
- `"iban"`: International Bank Account Number.
|
|
42
|
+
- `"mrn"`: Medical Record Number (may not work correctly, depending on provider and country).
|
|
43
|
+
- `"icd10"`: International Classification of Diseases, Tenth Revision.
|
|
44
|
+
- `"geo_coords"`: Geo-coordinates (latitude and longitude in decimal degrees format).
|
|
45
|
+
- `"username"`: Username handles (@username).
|
|
46
|
+
- `"file_path"`: File paths (general patterns for both Windows and Unix paths).
|
|
47
|
+
- `"bitcoin_wallet"`: Cryptocurrency wallet address.
|
|
48
|
+
- `"ethereum_wallet"`: Cryptocurrency wallet addresses.
|
|
49
|
+
|
|
36
50
|
## Changes in 0.2.2
|
|
37
51
|
|
|
38
52
|
We have refactored and changed the `detect_profanity` function:
|
|
@@ -138,6 +152,9 @@ from valx import detect_sensitive_information
|
|
|
138
152
|
detected_sensitive_info = detect_sensitive_information(sample_text)
|
|
139
153
|
```
|
|
140
154
|
|
|
155
|
+
> [!NOTE]
|
|
156
|
+
> We have updated this function, and it now includes an optional argument for `info_type`, which can be used to detect only specific types of sensitive information. It was also added to `remove_sensitive_information`.
|
|
157
|
+
|
|
141
158
|
### Remove Sensitive Information
|
|
142
159
|
|
|
143
160
|
```python
|
|
@@ -5,12 +5,26 @@
|
|
|
5
5
|

|
|
6
6
|

|
|
7
7
|
|
|
8
|
-
An open-source Python library for data cleaning tasks.
|
|
8
|
+
An open-source Python library for data cleaning tasks. It includes functions for profanity detection, and removal, and detection and removal of personal information. Also includes hate speech and offensive language detection and removal, using AI.
|
|
9
9
|
|
|
10
10
|
> [!IMPORTANT]
|
|
11
11
|
> Please downgrade to `numpy` version `1.26.4`. Our ValX **DecisionTreeClassifier** AI model, relies on lower versions of `numpy`, because it was trained on these versions.
|
|
12
12
|
> For more information see: https://techoverflow.net/2024/07/23/how-to-fix-numpy-dtype-size-changed-may-indicate-binary-incompatibility-expected-96-from-c-header-got-88-from-pyobject/
|
|
13
13
|
|
|
14
|
+
## Changes in 0.2.3
|
|
15
|
+
|
|
16
|
+
We have introduced a new optional `info_type` parameter into our `detect_sensitive_information`, and `remove_sensitive_information` functions, to allow you to have fine-grained control over what sensitive information you want to detect or remove.
|
|
17
|
+
|
|
18
|
+
Also introduced more detection patterns for other types of sensitive information, including:
|
|
19
|
+
- `"iban"`: International Bank Account Number.
|
|
20
|
+
- `"mrn"`: Medical Record Number (may not work correctly, depending on provider and country).
|
|
21
|
+
- `"icd10"`: International Classification of Diseases, Tenth Revision.
|
|
22
|
+
- `"geo_coords"`: Geo-coordinates (latitude and longitude in decimal degrees format).
|
|
23
|
+
- `"username"`: Username handles (@username).
|
|
24
|
+
- `"file_path"`: File paths (general patterns for both Windows and Unix paths).
|
|
25
|
+
- `"bitcoin_wallet"`: Cryptocurrency wallet address.
|
|
26
|
+
- `"ethereum_wallet"`: Cryptocurrency wallet addresses.
|
|
27
|
+
|
|
14
28
|
## Changes in 0.2.2
|
|
15
29
|
|
|
16
30
|
We have refactored and changed the `detect_profanity` function:
|
|
@@ -116,6 +130,9 @@ from valx import detect_sensitive_information
|
|
|
116
130
|
detected_sensitive_info = detect_sensitive_information(sample_text)
|
|
117
131
|
```
|
|
118
132
|
|
|
133
|
+
> [!NOTE]
|
|
134
|
+
> We have updated this function, and it now includes an optional argument for `info_type`, which can be used to detect only specific types of sensitive information. It was also added to `remove_sensitive_information`.
|
|
135
|
+
|
|
119
136
|
### Remove Sensitive Information
|
|
120
137
|
|
|
121
138
|
```python
|
|
@@ -5,7 +5,7 @@ with open("README.md", "r", encoding="utf-8") as fh:
|
|
|
5
5
|
|
|
6
6
|
setup(
|
|
7
7
|
name='valx',
|
|
8
|
-
version='0.2.
|
|
8
|
+
version='0.2.3',
|
|
9
9
|
author='Infinitode Pty Ltd',
|
|
10
10
|
author_email='infinitode.ltd@gmail.com',
|
|
11
11
|
description='An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.',
|
|
@@ -2825,85 +2825,68 @@ def remove_profanity(text_data, output_file=None, language='English'):
|
|
|
2825
2825
|
|
|
2826
2826
|
return cleaned_data
|
|
2827
2827
|
|
|
2828
|
-
|
|
2829
|
-
def detect_sensitive_information(text_data):
|
|
2828
|
+
def detect_sensitive_information(text_data, info_type=["email", "phone", "credit_card", "ssn", "id", "address", "ip", "iban", "mrn", "icd10", "geo_coords", "username", "file_path", "bitcoin_wallet", "ethereum_wallet"]):
|
|
2830
2829
|
"""
|
|
2831
2830
|
Detect sensitive information patterns in the provided text data.
|
|
2832
2831
|
|
|
2833
2832
|
Args:
|
|
2834
2833
|
text_data (list of str): A list of strings representing the text data to be analyzed.
|
|
2834
|
+
info_type (str or list of str, optional): One or more types of sensitive info to detect. Available types are: "email", "phone", "credit_card", "ssn", "id", "address", "ip", "iban", "mrn", "icd10", "geo_coords", "username", "file_path", "bitcoin_wallet", "ethereum_wallet". Uses all info types by default.
|
|
2835
2835
|
|
|
2836
2836
|
Returns:
|
|
2837
2837
|
list of tuple: A list of tuples containing detected sensitive information, each tuple
|
|
2838
2838
|
representing (line number, column index, type, value).
|
|
2839
2839
|
"""
|
|
2840
|
+
if isinstance(info_type, str):
|
|
2841
|
+
info_type = [info_type] # Convert single string to list for consistent handling
|
|
2842
|
+
|
|
2843
|
+
# Dictionary to map info_type to regex patterns
|
|
2844
|
+
regex_patterns = {
|
|
2845
|
+
"email": r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b',
|
|
2846
|
+
"phone": r'\b(?:\d{3}[-.\s]??\d{3}[-.\s]??\d{4}|\(\d{3}\)\s*\d{3}[-.\s]??\d{4}|\d{3}[-.\s]??\d{4})\b',
|
|
2847
|
+
"credit_card": r'\b(?:\d[ -]*?){13,16}\b',
|
|
2848
|
+
"ssn": r'\b\d{3}[-]?\d{2}[-]?\d{4}\b',
|
|
2849
|
+
"id": r'\b[A-Za-z]{1,2}\d{6,9}\b',
|
|
2850
|
+
"address": r'\b\d+\s\w+\s\w+,\s\w+,\s\w+\b',
|
|
2851
|
+
"ip": r'\b(?:[0-9]{1,3}\.){3}[0-9]{1,3}\b',
|
|
2852
|
+
"iban": r'\b[A-Z]{2}[0-9]{2}[A-Z0-9]{1,30}\b',
|
|
2853
|
+
"mrn": r'\b\d{5,10}\b',
|
|
2854
|
+
"icd10": r'\b[A-TV-Z][0-9]{2}(?:\.[0-9A-TV-Z]{1,4})?\b',
|
|
2855
|
+
"geo_coords": r'\b[-+]?([1-8]?\d(\.\d+)?|90(\.0+)?),\s*[-+]?(180(\.0+)?|((1[0-7]\d)|([1-9]?\d))(\.\d+)?)\b',
|
|
2856
|
+
"username": r'@\w+',
|
|
2857
|
+
"file_path": r'([A-Za-z]:\\|/)?([-\w\s]+\\|/)*[-\w\s]+\.\w+',
|
|
2858
|
+
"bitcoin_wallet": r'\b[13][a-km-zA-HJ-NP-Z1-9]{25,34}\b',
|
|
2859
|
+
"ethereum_wallet": r'\b0x[a-fA-F0-9]{40}\b'
|
|
2860
|
+
}
|
|
2861
|
+
|
|
2840
2862
|
sensitive_info = []
|
|
2841
2863
|
|
|
2842
|
-
#
|
|
2843
|
-
email_regex = r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b'
|
|
2844
|
-
# Regular expression for detecting phone numbers
|
|
2845
|
-
phone_regex = r'\b(?:\d{3}[-.\s]??\d{3}[-.\s]??\d{4}|\(\d{3}\)\s*\d{3}[-.\s]??\d{4}|\d{3}[-.\s]??\d{4})\b'
|
|
2846
|
-
# Regular expression for detecting credit card numbers (Visa, Mastercard, American Express, Discover)
|
|
2847
|
-
credit_card_regex = r'\b(?:\d[ -]*?){13,16}\b'
|
|
2848
|
-
# Regular expression for detecting social security numbers (SSN)
|
|
2849
|
-
ssn_regex = r'\b\d{3}[-]?\d{2}[-]?\d{4}\b'
|
|
2850
|
-
# Regular expression for detecting ID numbers (e.g., driver's license, passport)
|
|
2851
|
-
id_number_regex = r'\b[A-Za-z]{1,2}\d{6,9}\b'
|
|
2852
|
-
# Regular expression for detecting addresses (simple example)
|
|
2853
|
-
address_regex = r'\b\d+\s\w+\s\w+,\s\w+,\s\w+\b'
|
|
2854
|
-
# Regular expression for detecting IP addresses
|
|
2855
|
-
ip_address_regex = r'\b(?:[0-9]{1,3}\.){3}[0-9]{1,3}\b'
|
|
2856
|
-
|
|
2864
|
+
# Loop through each line and check for sensitive info types based on specified `info_type`
|
|
2857
2865
|
for i, line in enumerate(text_data):
|
|
2858
|
-
|
|
2859
|
-
|
|
2860
|
-
|
|
2861
|
-
|
|
2862
|
-
|
|
2863
|
-
|
|
2864
|
-
|
|
2865
|
-
for phone in phones:
|
|
2866
|
-
sensitive_info.append((i + 1, line.find(phone), 'Phone', phone))
|
|
2867
|
-
|
|
2868
|
-
# Detect credit card numbers
|
|
2869
|
-
credit_cards = re.findall(credit_card_regex, line)
|
|
2870
|
-
for card in credit_cards:
|
|
2871
|
-
sensitive_info.append((i + 1, line.find(card), 'Credit Card', card))
|
|
2872
|
-
|
|
2873
|
-
# Detect social security numbers
|
|
2874
|
-
ssn_numbers = re.findall(ssn_regex, line)
|
|
2875
|
-
for ssn in ssn_numbers:
|
|
2876
|
-
sensitive_info.append((i + 1, line.find(ssn), 'SSN', ssn))
|
|
2877
|
-
|
|
2878
|
-
# Detect ID numbers
|
|
2879
|
-
id_numbers = re.findall(id_number_regex, line)
|
|
2880
|
-
for id_number in id_numbers:
|
|
2881
|
-
sensitive_info.append((i + 1, line.find(id_number), 'ID Number', id_number))
|
|
2882
|
-
|
|
2883
|
-
# Detect addresses
|
|
2884
|
-
addresses = re.findall(address_regex, line)
|
|
2885
|
-
for address in addresses:
|
|
2886
|
-
sensitive_info.append((i + 1, line.find(address), 'Address', address))
|
|
2887
|
-
|
|
2888
|
-
# Detect IP addresses
|
|
2889
|
-
ip_addresses = re.findall(ip_address_regex, line)
|
|
2890
|
-
for ip_address in ip_addresses:
|
|
2891
|
-
sensitive_info.append((i + 1, line.find(ip_address), 'IP Address', ip_address))
|
|
2892
|
-
|
|
2866
|
+
for type_ in info_type:
|
|
2867
|
+
pattern = regex_patterns.get(type_)
|
|
2868
|
+
if pattern:
|
|
2869
|
+
matches = re.finditer(pattern, line)
|
|
2870
|
+
for match in matches:
|
|
2871
|
+
sensitive_info.append((i + 1, match.start(), type_.capitalize(), match.group()))
|
|
2872
|
+
|
|
2893
2873
|
return sensitive_info
|
|
2894
2874
|
|
|
2895
|
-
def remove_sensitive_information(text_data, output_file=None):
|
|
2875
|
+
def remove_sensitive_information(text_data, output_file=None, info_type=["email", "phone", "credit_card", "ssn", "id", "address", "ip", "iban", "mrn", "icd10", "geo_coords", "username", "file_path", "bitcoin_wallet", "ethereum_wallet"]):
|
|
2896
2876
|
"""
|
|
2897
2877
|
Remove sensitive information patterns from the provided text data.
|
|
2898
2878
|
|
|
2899
2879
|
Args:
|
|
2900
2880
|
text_data (list of str): A list of strings representing the text data to be cleaned.
|
|
2901
2881
|
output_file (str, optional): Path to the output file where cleaned data will be saved.
|
|
2882
|
+
info_type (str or list of str, optional): One or more types of sensitive info to detect and remove. Available types are: "email", "phone", "credit_card", "ssn", "id", "address", "ip", "iban", "mrn", "icd10", "geo_coords", "username", "file_path", "bitcoin_wallet", "ethereum_wallet". Uses all info types by default.
|
|
2902
2883
|
|
|
2903
2884
|
Returns:
|
|
2904
2885
|
list of str: A list of strings representing the cleaned text data.
|
|
2905
2886
|
"""
|
|
2906
|
-
|
|
2887
|
+
# Detect sensitive information
|
|
2888
|
+
sensitive_info = detect_sensitive_information(text_data, info_type=info_type)
|
|
2889
|
+
|
|
2907
2890
|
cleaned_data = []
|
|
2908
2891
|
for line in text_data:
|
|
2909
2892
|
cleaned_line = line
|
|
@@ -2917,7 +2900,6 @@ def remove_sensitive_information(text_data, output_file=None):
|
|
|
2917
2900
|
file.write('\n'.join(cleaned_data))
|
|
2918
2901
|
return cleaned_data
|
|
2919
2902
|
|
|
2920
|
-
|
|
2921
2903
|
# New version
|
|
2922
2904
|
MODEL_DIR = os.path.join(os.path.dirname(__file__), 'models')
|
|
2923
2905
|
DECISION_TREE_MODEL_PATH = os.path.join(MODEL_DIR, 'decision_tree_model.sav')
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.1
|
|
2
2
|
Name: valx
|
|
3
|
-
Version: 0.2.
|
|
3
|
+
Version: 0.2.3
|
|
4
4
|
Summary: An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.
|
|
5
5
|
Home-page: https://github.com/infinitode/valx
|
|
6
6
|
Author: Infinitode Pty Ltd
|
|
@@ -27,12 +27,26 @@ Requires-Dist: scikit-learn==1.2.2
|
|
|
27
27
|

|
|
28
28
|

|
|
29
29
|
|
|
30
|
-
An open-source Python library for data cleaning tasks.
|
|
30
|
+
An open-source Python library for data cleaning tasks. It includes functions for profanity detection, and removal, and detection and removal of personal information. Also includes hate speech and offensive language detection and removal, using AI.
|
|
31
31
|
|
|
32
32
|
> [!IMPORTANT]
|
|
33
33
|
> Please downgrade to `numpy` version `1.26.4`. Our ValX **DecisionTreeClassifier** AI model, relies on lower versions of `numpy`, because it was trained on these versions.
|
|
34
34
|
> For more information see: https://techoverflow.net/2024/07/23/how-to-fix-numpy-dtype-size-changed-may-indicate-binary-incompatibility-expected-96-from-c-header-got-88-from-pyobject/
|
|
35
35
|
|
|
36
|
+
## Changes in 0.2.3
|
|
37
|
+
|
|
38
|
+
We have introduced a new optional `info_type` parameter into our `detect_sensitive_information`, and `remove_sensitive_information` functions, to allow you to have fine-grained control over what sensitive information you want to detect or remove.
|
|
39
|
+
|
|
40
|
+
Also introduced more detection patterns for other types of sensitive information, including:
|
|
41
|
+
- `"iban"`: International Bank Account Number.
|
|
42
|
+
- `"mrn"`: Medical Record Number (may not work correctly, depending on provider and country).
|
|
43
|
+
- `"icd10"`: International Classification of Diseases, Tenth Revision.
|
|
44
|
+
- `"geo_coords"`: Geo-coordinates (latitude and longitude in decimal degrees format).
|
|
45
|
+
- `"username"`: Username handles (@username).
|
|
46
|
+
- `"file_path"`: File paths (general patterns for both Windows and Unix paths).
|
|
47
|
+
- `"bitcoin_wallet"`: Cryptocurrency wallet address.
|
|
48
|
+
- `"ethereum_wallet"`: Cryptocurrency wallet addresses.
|
|
49
|
+
|
|
36
50
|
## Changes in 0.2.2
|
|
37
51
|
|
|
38
52
|
We have refactored and changed the `detect_profanity` function:
|
|
@@ -138,6 +152,9 @@ from valx import detect_sensitive_information
|
|
|
138
152
|
detected_sensitive_info = detect_sensitive_information(sample_text)
|
|
139
153
|
```
|
|
140
154
|
|
|
155
|
+
> [!NOTE]
|
|
156
|
+
> We have updated this function, and it now includes an optional argument for `info_type`, which can be used to detect only specific types of sensitive information. It was also added to `remove_sensitive_information`.
|
|
157
|
+
|
|
141
158
|
### Remove Sensitive Information
|
|
142
159
|
|
|
143
160
|
```python
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|