valx 0.2.2__tar.gz → 0.2.4__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {valx-0.2.2/valx.egg-info → valx-0.2.4}/PKG-INFO +27 -4
- {valx-0.2.2 → valx-0.2.4}/README.md +24 -1
- {valx-0.2.2 → valx-0.2.4}/setup.py +3 -3
- {valx-0.2.2 → valx-0.2.4}/valx/__init__.py +49 -56
- valx-0.2.4/valx/models/classifier_upgraded.pkl +0 -0
- {valx-0.2.2 → valx-0.2.4/valx.egg-info}/PKG-INFO +27 -4
- {valx-0.2.2 → valx-0.2.4}/valx.egg-info/SOURCES.txt +1 -0
- valx-0.2.4/valx.egg-info/requires.txt +1 -0
- valx-0.2.2/valx.egg-info/requires.txt +0 -1
- {valx-0.2.2 → valx-0.2.4}/LICENSE +0 -0
- {valx-0.2.2 → valx-0.2.4}/setup.cfg +0 -0
- {valx-0.2.2 → valx-0.2.4}/valx/models/count_vectorizer.sav +0 -0
- {valx-0.2.2 → valx-0.2.4}/valx/models/decision_tree_model.sav +0 -0
- {valx-0.2.2 → valx-0.2.4}/valx.egg-info/dependency_links.txt +0 -0
- {valx-0.2.2 → valx-0.2.4}/valx.egg-info/top_level.txt +0 -0
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
Metadata-Version: 2.1
|
|
2
2
|
Name: valx
|
|
3
|
-
Version: 0.2.
|
|
3
|
+
Version: 0.2.4
|
|
4
4
|
Summary: An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.
|
|
5
5
|
Home-page: https://github.com/infinitode/valx
|
|
6
6
|
Author: Infinitode Pty Ltd
|
|
7
7
|
Author-email: infinitode.ltd@gmail.com
|
|
8
|
-
Classifier: Development Status ::
|
|
8
|
+
Classifier: Development Status :: 5 - Production/Stable
|
|
9
9
|
Classifier: Intended Audience :: Developers
|
|
10
10
|
Classifier: License :: OSI Approved :: MIT License
|
|
11
11
|
Classifier: Programming Language :: Python :: 3.6
|
|
@@ -18,7 +18,7 @@ Classifier: Programming Language :: Python :: 3.12
|
|
|
18
18
|
Requires-Python: >=3.6
|
|
19
19
|
Description-Content-Type: text/markdown
|
|
20
20
|
License-File: LICENSE
|
|
21
|
-
Requires-Dist: scikit-learn
|
|
21
|
+
Requires-Dist: scikit-learn
|
|
22
22
|
|
|
23
23
|
# ValX
|
|
24
24
|

|
|
@@ -27,12 +27,32 @@ Requires-Dist: scikit-learn==1.2.2
|
|
|
27
27
|

|
|
28
28
|

|
|
29
29
|
|
|
30
|
-
An open-source Python library for data cleaning tasks.
|
|
30
|
+
An open-source Python library for data cleaning tasks. It includes functions for profanity detection, and removal, and detection and removal of personal information. Also includes hate speech and offensive language detection and removal, using AI.
|
|
31
31
|
|
|
32
32
|
> [!IMPORTANT]
|
|
33
33
|
> Please downgrade to `numpy` version `1.26.4`. Our ValX **DecisionTreeClassifier** AI model, relies on lower versions of `numpy`, because it was trained on these versions.
|
|
34
34
|
> For more information see: https://techoverflow.net/2024/07/23/how-to-fix-numpy-dtype-size-changed-may-indicate-binary-incompatibility-expected-96-from-c-header-got-88-from-pyobject/
|
|
35
35
|
|
|
36
|
+
## Changes in 0.2.4
|
|
37
|
+
|
|
38
|
+
Fixed a major incompatibility issue with `scikit-learn` due to version changes in `scikit-learn v1.3.0` which causes compatibility issues with versions later than `1.2.2`. ValX can now be used with `scikit-learn` versions earlier and later than `1.3.0`!
|
|
39
|
+
|
|
40
|
+
We've also removed `scikit-learn==1.2.2` as a dependency, as most versions of `scikit-learn` will now work.
|
|
41
|
+
|
|
42
|
+
## Changes in 0.2.3
|
|
43
|
+
|
|
44
|
+
We have introduced a new optional `info_type` parameter into our `detect_sensitive_information`, and `remove_sensitive_information` functions, to allow you to have fine-grained control over what sensitive information you want to detect or remove.
|
|
45
|
+
|
|
46
|
+
Also introduced more detection patterns for other types of sensitive information, including:
|
|
47
|
+
- `"iban"`: International Bank Account Number.
|
|
48
|
+
- `"mrn"`: Medical Record Number (may not work correctly, depending on provider and country).
|
|
49
|
+
- `"icd10"`: International Classification of Diseases, Tenth Revision.
|
|
50
|
+
- `"geo_coords"`: Geo-coordinates (latitude and longitude in decimal degrees format).
|
|
51
|
+
- `"username"`: Username handles (@username).
|
|
52
|
+
- `"file_path"`: File paths (general patterns for both Windows and Unix paths).
|
|
53
|
+
- `"bitcoin_wallet"`: Cryptocurrency wallet address.
|
|
54
|
+
- `"ethereum_wallet"`: Cryptocurrency wallet addresses.
|
|
55
|
+
|
|
36
56
|
## Changes in 0.2.2
|
|
37
57
|
|
|
38
58
|
We have refactored and changed the `detect_profanity` function:
|
|
@@ -138,6 +158,9 @@ from valx import detect_sensitive_information
|
|
|
138
158
|
detected_sensitive_info = detect_sensitive_information(sample_text)
|
|
139
159
|
```
|
|
140
160
|
|
|
161
|
+
> [!NOTE]
|
|
162
|
+
> We have updated this function, and it now includes an optional argument for `info_type`, which can be used to detect only specific types of sensitive information. It was also added to `remove_sensitive_information`.
|
|
163
|
+
|
|
141
164
|
### Remove Sensitive Information
|
|
142
165
|
|
|
143
166
|
```python
|
|
@@ -5,12 +5,32 @@
|
|
|
5
5
|

|
|
6
6
|

|
|
7
7
|
|
|
8
|
-
An open-source Python library for data cleaning tasks.
|
|
8
|
+
An open-source Python library for data cleaning tasks. It includes functions for profanity detection, and removal, and detection and removal of personal information. Also includes hate speech and offensive language detection and removal, using AI.
|
|
9
9
|
|
|
10
10
|
> [!IMPORTANT]
|
|
11
11
|
> Please downgrade to `numpy` version `1.26.4`. Our ValX **DecisionTreeClassifier** AI model, relies on lower versions of `numpy`, because it was trained on these versions.
|
|
12
12
|
> For more information see: https://techoverflow.net/2024/07/23/how-to-fix-numpy-dtype-size-changed-may-indicate-binary-incompatibility-expected-96-from-c-header-got-88-from-pyobject/
|
|
13
13
|
|
|
14
|
+
## Changes in 0.2.4
|
|
15
|
+
|
|
16
|
+
Fixed a major incompatibility issue with `scikit-learn` due to version changes in `scikit-learn v1.3.0` which causes compatibility issues with versions later than `1.2.2`. ValX can now be used with `scikit-learn` versions earlier and later than `1.3.0`!
|
|
17
|
+
|
|
18
|
+
We've also removed `scikit-learn==1.2.2` as a dependency, as most versions of `scikit-learn` will now work.
|
|
19
|
+
|
|
20
|
+
## Changes in 0.2.3
|
|
21
|
+
|
|
22
|
+
We have introduced a new optional `info_type` parameter into our `detect_sensitive_information`, and `remove_sensitive_information` functions, to allow you to have fine-grained control over what sensitive information you want to detect or remove.
|
|
23
|
+
|
|
24
|
+
Also introduced more detection patterns for other types of sensitive information, including:
|
|
25
|
+
- `"iban"`: International Bank Account Number.
|
|
26
|
+
- `"mrn"`: Medical Record Number (may not work correctly, depending on provider and country).
|
|
27
|
+
- `"icd10"`: International Classification of Diseases, Tenth Revision.
|
|
28
|
+
- `"geo_coords"`: Geo-coordinates (latitude and longitude in decimal degrees format).
|
|
29
|
+
- `"username"`: Username handles (@username).
|
|
30
|
+
- `"file_path"`: File paths (general patterns for both Windows and Unix paths).
|
|
31
|
+
- `"bitcoin_wallet"`: Cryptocurrency wallet address.
|
|
32
|
+
- `"ethereum_wallet"`: Cryptocurrency wallet addresses.
|
|
33
|
+
|
|
14
34
|
## Changes in 0.2.2
|
|
15
35
|
|
|
16
36
|
We have refactored and changed the `detect_profanity` function:
|
|
@@ -116,6 +136,9 @@ from valx import detect_sensitive_information
|
|
|
116
136
|
detected_sensitive_info = detect_sensitive_information(sample_text)
|
|
117
137
|
```
|
|
118
138
|
|
|
139
|
+
> [!NOTE]
|
|
140
|
+
> We have updated this function, and it now includes an optional argument for `info_type`, which can be used to detect only specific types of sensitive information. It was also added to `remove_sensitive_information`.
|
|
141
|
+
|
|
119
142
|
### Remove Sensitive Information
|
|
120
143
|
|
|
121
144
|
```python
|
|
@@ -5,7 +5,7 @@ with open("README.md", "r", encoding="utf-8") as fh:
|
|
|
5
5
|
|
|
6
6
|
setup(
|
|
7
7
|
name='valx',
|
|
8
|
-
version='0.2.
|
|
8
|
+
version='0.2.4',
|
|
9
9
|
author='Infinitode Pty Ltd',
|
|
10
10
|
author_email='infinitode.ltd@gmail.com',
|
|
11
11
|
description='An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.',
|
|
@@ -15,10 +15,10 @@ setup(
|
|
|
15
15
|
packages=find_packages(),
|
|
16
16
|
package_data={'valx': ['models/*']},
|
|
17
17
|
install_requires=[
|
|
18
|
-
'scikit-learn
|
|
18
|
+
'scikit-learn'
|
|
19
19
|
],
|
|
20
20
|
classifiers=[
|
|
21
|
-
'Development Status ::
|
|
21
|
+
'Development Status :: 5 - Production/Stable',
|
|
22
22
|
'Intended Audience :: Developers',
|
|
23
23
|
'License :: OSI Approved :: MIT License',
|
|
24
24
|
'Programming Language :: Python :: 3.6',
|
|
@@ -3,6 +3,7 @@ import re
|
|
|
3
3
|
|
|
4
4
|
# New version, that includes AI detection
|
|
5
5
|
import pickle
|
|
6
|
+
from sklearn import __version__ as sklearnversion
|
|
6
7
|
from sklearn.tree import DecisionTreeClassifier # Import the DecisionTreeClassifier class
|
|
7
8
|
from sklearn.feature_extraction.text import CountVectorizer # Import the CountVectorizer class
|
|
8
9
|
|
|
@@ -2825,85 +2826,68 @@ def remove_profanity(text_data, output_file=None, language='English'):
|
|
|
2825
2826
|
|
|
2826
2827
|
return cleaned_data
|
|
2827
2828
|
|
|
2828
|
-
|
|
2829
|
-
def detect_sensitive_information(text_data):
|
|
2829
|
+
def detect_sensitive_information(text_data, info_type=["email", "phone", "credit_card", "ssn", "id", "address", "ip", "iban", "mrn", "icd10", "geo_coords", "username", "file_path", "bitcoin_wallet", "ethereum_wallet"]):
|
|
2830
2830
|
"""
|
|
2831
2831
|
Detect sensitive information patterns in the provided text data.
|
|
2832
2832
|
|
|
2833
2833
|
Args:
|
|
2834
2834
|
text_data (list of str): A list of strings representing the text data to be analyzed.
|
|
2835
|
+
info_type (str or list of str, optional): One or more types of sensitive info to detect. Available types are: "email", "phone", "credit_card", "ssn", "id", "address", "ip", "iban", "mrn", "icd10", "geo_coords", "username", "file_path", "bitcoin_wallet", "ethereum_wallet". Uses all info types by default.
|
|
2835
2836
|
|
|
2836
2837
|
Returns:
|
|
2837
2838
|
list of tuple: A list of tuples containing detected sensitive information, each tuple
|
|
2838
2839
|
representing (line number, column index, type, value).
|
|
2839
2840
|
"""
|
|
2841
|
+
if isinstance(info_type, str):
|
|
2842
|
+
info_type = [info_type] # Convert single string to list for consistent handling
|
|
2843
|
+
|
|
2844
|
+
# Dictionary to map info_type to regex patterns
|
|
2845
|
+
regex_patterns = {
|
|
2846
|
+
"email": r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b',
|
|
2847
|
+
"phone": r'\b(?:\d{3}[-.\s]??\d{3}[-.\s]??\d{4}|\(\d{3}\)\s*\d{3}[-.\s]??\d{4}|\d{3}[-.\s]??\d{4})\b',
|
|
2848
|
+
"credit_card": r'\b(?:\d[ -]*?){13,16}\b',
|
|
2849
|
+
"ssn": r'\b\d{3}[-]?\d{2}[-]?\d{4}\b',
|
|
2850
|
+
"id": r'\b[A-Za-z]{1,2}\d{6,9}\b',
|
|
2851
|
+
"address": r'\b\d+\s\w+\s\w+,\s\w+,\s\w+\b',
|
|
2852
|
+
"ip": r'\b(?:[0-9]{1,3}\.){3}[0-9]{1,3}\b',
|
|
2853
|
+
"iban": r'\b[A-Z]{2}[0-9]{2}[A-Z0-9]{1,30}\b',
|
|
2854
|
+
"mrn": r'\b\d{5,10}\b',
|
|
2855
|
+
"icd10": r'\b[A-TV-Z][0-9]{2}(?:\.[0-9A-TV-Z]{1,4})?\b',
|
|
2856
|
+
"geo_coords": r'\b[-+]?([1-8]?\d(\.\d+)?|90(\.0+)?),\s*[-+]?(180(\.0+)?|((1[0-7]\d)|([1-9]?\d))(\.\d+)?)\b',
|
|
2857
|
+
"username": r'@\w+',
|
|
2858
|
+
"file_path": r'([A-Za-z]:\\|/)?([-\w\s]+\\|/)*[-\w\s]+\.\w+',
|
|
2859
|
+
"bitcoin_wallet": r'\b[13][a-km-zA-HJ-NP-Z1-9]{25,34}\b',
|
|
2860
|
+
"ethereum_wallet": r'\b0x[a-fA-F0-9]{40}\b'
|
|
2861
|
+
}
|
|
2862
|
+
|
|
2840
2863
|
sensitive_info = []
|
|
2841
2864
|
|
|
2842
|
-
#
|
|
2843
|
-
email_regex = r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b'
|
|
2844
|
-
# Regular expression for detecting phone numbers
|
|
2845
|
-
phone_regex = r'\b(?:\d{3}[-.\s]??\d{3}[-.\s]??\d{4}|\(\d{3}\)\s*\d{3}[-.\s]??\d{4}|\d{3}[-.\s]??\d{4})\b'
|
|
2846
|
-
# Regular expression for detecting credit card numbers (Visa, Mastercard, American Express, Discover)
|
|
2847
|
-
credit_card_regex = r'\b(?:\d[ -]*?){13,16}\b'
|
|
2848
|
-
# Regular expression for detecting social security numbers (SSN)
|
|
2849
|
-
ssn_regex = r'\b\d{3}[-]?\d{2}[-]?\d{4}\b'
|
|
2850
|
-
# Regular expression for detecting ID numbers (e.g., driver's license, passport)
|
|
2851
|
-
id_number_regex = r'\b[A-Za-z]{1,2}\d{6,9}\b'
|
|
2852
|
-
# Regular expression for detecting addresses (simple example)
|
|
2853
|
-
address_regex = r'\b\d+\s\w+\s\w+,\s\w+,\s\w+\b'
|
|
2854
|
-
# Regular expression for detecting IP addresses
|
|
2855
|
-
ip_address_regex = r'\b(?:[0-9]{1,3}\.){3}[0-9]{1,3}\b'
|
|
2856
|
-
|
|
2865
|
+
# Loop through each line and check for sensitive info types based on specified `info_type`
|
|
2857
2866
|
for i, line in enumerate(text_data):
|
|
2858
|
-
|
|
2859
|
-
|
|
2860
|
-
|
|
2861
|
-
|
|
2862
|
-
|
|
2863
|
-
|
|
2864
|
-
|
|
2865
|
-
for phone in phones:
|
|
2866
|
-
sensitive_info.append((i + 1, line.find(phone), 'Phone', phone))
|
|
2867
|
-
|
|
2868
|
-
# Detect credit card numbers
|
|
2869
|
-
credit_cards = re.findall(credit_card_regex, line)
|
|
2870
|
-
for card in credit_cards:
|
|
2871
|
-
sensitive_info.append((i + 1, line.find(card), 'Credit Card', card))
|
|
2872
|
-
|
|
2873
|
-
# Detect social security numbers
|
|
2874
|
-
ssn_numbers = re.findall(ssn_regex, line)
|
|
2875
|
-
for ssn in ssn_numbers:
|
|
2876
|
-
sensitive_info.append((i + 1, line.find(ssn), 'SSN', ssn))
|
|
2877
|
-
|
|
2878
|
-
# Detect ID numbers
|
|
2879
|
-
id_numbers = re.findall(id_number_regex, line)
|
|
2880
|
-
for id_number in id_numbers:
|
|
2881
|
-
sensitive_info.append((i + 1, line.find(id_number), 'ID Number', id_number))
|
|
2882
|
-
|
|
2883
|
-
# Detect addresses
|
|
2884
|
-
addresses = re.findall(address_regex, line)
|
|
2885
|
-
for address in addresses:
|
|
2886
|
-
sensitive_info.append((i + 1, line.find(address), 'Address', address))
|
|
2887
|
-
|
|
2888
|
-
# Detect IP addresses
|
|
2889
|
-
ip_addresses = re.findall(ip_address_regex, line)
|
|
2890
|
-
for ip_address in ip_addresses:
|
|
2891
|
-
sensitive_info.append((i + 1, line.find(ip_address), 'IP Address', ip_address))
|
|
2892
|
-
|
|
2867
|
+
for type_ in info_type:
|
|
2868
|
+
pattern = regex_patterns.get(type_)
|
|
2869
|
+
if pattern:
|
|
2870
|
+
matches = re.finditer(pattern, line)
|
|
2871
|
+
for match in matches:
|
|
2872
|
+
sensitive_info.append((i + 1, match.start(), type_.capitalize(), match.group()))
|
|
2873
|
+
|
|
2893
2874
|
return sensitive_info
|
|
2894
2875
|
|
|
2895
|
-
def remove_sensitive_information(text_data, output_file=None):
|
|
2876
|
+
def remove_sensitive_information(text_data, output_file=None, info_type=["email", "phone", "credit_card", "ssn", "id", "address", "ip", "iban", "mrn", "icd10", "geo_coords", "username", "file_path", "bitcoin_wallet", "ethereum_wallet"]):
|
|
2896
2877
|
"""
|
|
2897
2878
|
Remove sensitive information patterns from the provided text data.
|
|
2898
2879
|
|
|
2899
2880
|
Args:
|
|
2900
2881
|
text_data (list of str): A list of strings representing the text data to be cleaned.
|
|
2901
2882
|
output_file (str, optional): Path to the output file where cleaned data will be saved.
|
|
2883
|
+
info_type (str or list of str, optional): One or more types of sensitive info to detect and remove. Available types are: "email", "phone", "credit_card", "ssn", "id", "address", "ip", "iban", "mrn", "icd10", "geo_coords", "username", "file_path", "bitcoin_wallet", "ethereum_wallet". Uses all info types by default.
|
|
2902
2884
|
|
|
2903
2885
|
Returns:
|
|
2904
2886
|
list of str: A list of strings representing the cleaned text data.
|
|
2905
2887
|
"""
|
|
2906
|
-
|
|
2888
|
+
# Detect sensitive information
|
|
2889
|
+
sensitive_info = detect_sensitive_information(text_data, info_type=info_type)
|
|
2890
|
+
|
|
2907
2891
|
cleaned_data = []
|
|
2908
2892
|
for line in text_data:
|
|
2909
2893
|
cleaned_line = line
|
|
@@ -2917,11 +2901,20 @@ def remove_sensitive_information(text_data, output_file=None):
|
|
|
2917
2901
|
file.write('\n'.join(cleaned_data))
|
|
2918
2902
|
return cleaned_data
|
|
2919
2903
|
|
|
2920
|
-
|
|
2921
2904
|
# New version
|
|
2922
2905
|
MODEL_DIR = os.path.join(os.path.dirname(__file__), 'models')
|
|
2923
|
-
DECISION_TREE_MODEL_PATH = os.path.join(MODEL_DIR, 'decision_tree_model.sav')
|
|
2924
2906
|
COUNT_VECTORIZER_PATH = os.path.join(MODEL_DIR, 'count_vectorizer.sav')
|
|
2907
|
+
DEFAULT_MODEL_FILENAME = "decision_tree_model.sav"
|
|
2908
|
+
UPGRADED_MODEL_FILENAME = "classifier_upgraded.pkl"
|
|
2909
|
+
|
|
2910
|
+
# Check scikit-learn version
|
|
2911
|
+
SKLEARN_VERSION = tuple(map(int, sklearnversion.split(".")))
|
|
2912
|
+
|
|
2913
|
+
# Set the model path based on the version
|
|
2914
|
+
if SKLEARN_VERSION >= (1, 3, 0):
|
|
2915
|
+
DECISION_TREE_MODEL_PATH = os.path.join(MODEL_DIR, UPGRADED_MODEL_FILENAME)
|
|
2916
|
+
else:
|
|
2917
|
+
DECISION_TREE_MODEL_PATH = os.path.join(MODEL_DIR, DEFAULT_MODEL_FILENAME)
|
|
2925
2918
|
|
|
2926
2919
|
# Load the saved models
|
|
2927
2920
|
model = pickle.load(open(DECISION_TREE_MODEL_PATH, 'rb'))
|
|
Binary file
|
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
Metadata-Version: 2.1
|
|
2
2
|
Name: valx
|
|
3
|
-
Version: 0.2.
|
|
3
|
+
Version: 0.2.4
|
|
4
4
|
Summary: An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.
|
|
5
5
|
Home-page: https://github.com/infinitode/valx
|
|
6
6
|
Author: Infinitode Pty Ltd
|
|
7
7
|
Author-email: infinitode.ltd@gmail.com
|
|
8
|
-
Classifier: Development Status ::
|
|
8
|
+
Classifier: Development Status :: 5 - Production/Stable
|
|
9
9
|
Classifier: Intended Audience :: Developers
|
|
10
10
|
Classifier: License :: OSI Approved :: MIT License
|
|
11
11
|
Classifier: Programming Language :: Python :: 3.6
|
|
@@ -18,7 +18,7 @@ Classifier: Programming Language :: Python :: 3.12
|
|
|
18
18
|
Requires-Python: >=3.6
|
|
19
19
|
Description-Content-Type: text/markdown
|
|
20
20
|
License-File: LICENSE
|
|
21
|
-
Requires-Dist: scikit-learn
|
|
21
|
+
Requires-Dist: scikit-learn
|
|
22
22
|
|
|
23
23
|
# ValX
|
|
24
24
|

|
|
@@ -27,12 +27,32 @@ Requires-Dist: scikit-learn==1.2.2
|
|
|
27
27
|

|
|
28
28
|

|
|
29
29
|
|
|
30
|
-
An open-source Python library for data cleaning tasks.
|
|
30
|
+
An open-source Python library for data cleaning tasks. It includes functions for profanity detection, and removal, and detection and removal of personal information. Also includes hate speech and offensive language detection and removal, using AI.
|
|
31
31
|
|
|
32
32
|
> [!IMPORTANT]
|
|
33
33
|
> Please downgrade to `numpy` version `1.26.4`. Our ValX **DecisionTreeClassifier** AI model, relies on lower versions of `numpy`, because it was trained on these versions.
|
|
34
34
|
> For more information see: https://techoverflow.net/2024/07/23/how-to-fix-numpy-dtype-size-changed-may-indicate-binary-incompatibility-expected-96-from-c-header-got-88-from-pyobject/
|
|
35
35
|
|
|
36
|
+
## Changes in 0.2.4
|
|
37
|
+
|
|
38
|
+
Fixed a major incompatibility issue with `scikit-learn` due to version changes in `scikit-learn v1.3.0` which causes compatibility issues with versions later than `1.2.2`. ValX can now be used with `scikit-learn` versions earlier and later than `1.3.0`!
|
|
39
|
+
|
|
40
|
+
We've also removed `scikit-learn==1.2.2` as a dependency, as most versions of `scikit-learn` will now work.
|
|
41
|
+
|
|
42
|
+
## Changes in 0.2.3
|
|
43
|
+
|
|
44
|
+
We have introduced a new optional `info_type` parameter into our `detect_sensitive_information`, and `remove_sensitive_information` functions, to allow you to have fine-grained control over what sensitive information you want to detect or remove.
|
|
45
|
+
|
|
46
|
+
Also introduced more detection patterns for other types of sensitive information, including:
|
|
47
|
+
- `"iban"`: International Bank Account Number.
|
|
48
|
+
- `"mrn"`: Medical Record Number (may not work correctly, depending on provider and country).
|
|
49
|
+
- `"icd10"`: International Classification of Diseases, Tenth Revision.
|
|
50
|
+
- `"geo_coords"`: Geo-coordinates (latitude and longitude in decimal degrees format).
|
|
51
|
+
- `"username"`: Username handles (@username).
|
|
52
|
+
- `"file_path"`: File paths (general patterns for both Windows and Unix paths).
|
|
53
|
+
- `"bitcoin_wallet"`: Cryptocurrency wallet address.
|
|
54
|
+
- `"ethereum_wallet"`: Cryptocurrency wallet addresses.
|
|
55
|
+
|
|
36
56
|
## Changes in 0.2.2
|
|
37
57
|
|
|
38
58
|
We have refactored and changed the `detect_profanity` function:
|
|
@@ -138,6 +158,9 @@ from valx import detect_sensitive_information
|
|
|
138
158
|
detected_sensitive_info = detect_sensitive_information(sample_text)
|
|
139
159
|
```
|
|
140
160
|
|
|
161
|
+
> [!NOTE]
|
|
162
|
+
> We have updated this function, and it now includes an optional argument for `info_type`, which can be used to detect only specific types of sensitive information. It was also added to `remove_sensitive_information`.
|
|
163
|
+
|
|
141
164
|
### Remove Sensitive Information
|
|
142
165
|
|
|
143
166
|
```python
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
scikit-learn
|
|
@@ -1 +0,0 @@
|
|
|
1
|
-
scikit-learn==1.2.2
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|