valx 0.2.2__tar.gz → 0.2.4__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,11 +1,11 @@
1
1
  Metadata-Version: 2.1
2
2
  Name: valx
3
- Version: 0.2.2
3
+ Version: 0.2.4
4
4
  Summary: An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.
5
5
  Home-page: https://github.com/infinitode/valx
6
6
  Author: Infinitode Pty Ltd
7
7
  Author-email: infinitode.ltd@gmail.com
8
- Classifier: Development Status :: 3 - Alpha
8
+ Classifier: Development Status :: 5 - Production/Stable
9
9
  Classifier: Intended Audience :: Developers
10
10
  Classifier: License :: OSI Approved :: MIT License
11
11
  Classifier: Programming Language :: Python :: 3.6
@@ -18,7 +18,7 @@ Classifier: Programming Language :: Python :: 3.12
18
18
  Requires-Python: >=3.6
19
19
  Description-Content-Type: text/markdown
20
20
  License-File: LICENSE
21
- Requires-Dist: scikit-learn==1.2.2
21
+ Requires-Dist: scikit-learn
22
22
 
23
23
  # ValX
24
24
  ![Python Version](https://img.shields.io/badge/python-3.12-blue.svg)
@@ -27,12 +27,32 @@ Requires-Dist: scikit-learn==1.2.2
27
27
  ![License Compliance](https://img.shields.io/badge/license-compliance-brightgreen.svg)
28
28
  ![PyPI Version](https://img.shields.io/pypi/v/valx)
29
29
 
30
- An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. It also now provides detection and removal of personal information. Now includes hate speech and offensive language detection using AI.
30
+ An open-source Python library for data cleaning tasks. It includes functions for profanity detection, and removal, and detection and removal of personal information. Also includes hate speech and offensive language detection and removal, using AI.
31
31
 
32
32
  > [!IMPORTANT]
33
33
  > Please downgrade to `numpy` version `1.26.4`. Our ValX **DecisionTreeClassifier** AI model, relies on lower versions of `numpy`, because it was trained on these versions.
34
34
  > For more information see: https://techoverflow.net/2024/07/23/how-to-fix-numpy-dtype-size-changed-may-indicate-binary-incompatibility-expected-96-from-c-header-got-88-from-pyobject/
35
35
 
36
+ ## Changes in 0.2.4
37
+
38
+ Fixed a major incompatibility issue with `scikit-learn` due to version changes in `scikit-learn v1.3.0` which causes compatibility issues with versions later than `1.2.2`. ValX can now be used with `scikit-learn` versions earlier and later than `1.3.0`!
39
+
40
+ We've also removed `scikit-learn==1.2.2` as a dependency, as most versions of `scikit-learn` will now work.
41
+
42
+ ## Changes in 0.2.3
43
+
44
+ We have introduced a new optional `info_type` parameter into our `detect_sensitive_information`, and `remove_sensitive_information` functions, to allow you to have fine-grained control over what sensitive information you want to detect or remove.
45
+
46
+ Also introduced more detection patterns for other types of sensitive information, including:
47
+ - `"iban"`: International Bank Account Number.
48
+ - `"mrn"`: Medical Record Number (may not work correctly, depending on provider and country).
49
+ - `"icd10"`: International Classification of Diseases, Tenth Revision.
50
+ - `"geo_coords"`: Geo-coordinates (latitude and longitude in decimal degrees format).
51
+ - `"username"`: Username handles (@username).
52
+ - `"file_path"`: File paths (general patterns for both Windows and Unix paths).
53
+ - `"bitcoin_wallet"`: Cryptocurrency wallet address.
54
+ - `"ethereum_wallet"`: Cryptocurrency wallet addresses.
55
+
36
56
  ## Changes in 0.2.2
37
57
 
38
58
  We have refactored and changed the `detect_profanity` function:
@@ -138,6 +158,9 @@ from valx import detect_sensitive_information
138
158
  detected_sensitive_info = detect_sensitive_information(sample_text)
139
159
  ```
140
160
 
161
+ > [!NOTE]
162
+ > We have updated this function, and it now includes an optional argument for `info_type`, which can be used to detect only specific types of sensitive information. It was also added to `remove_sensitive_information`.
163
+
141
164
  ### Remove Sensitive Information
142
165
 
143
166
  ```python
@@ -5,12 +5,32 @@
5
5
  ![License Compliance](https://img.shields.io/badge/license-compliance-brightgreen.svg)
6
6
  ![PyPI Version](https://img.shields.io/pypi/v/valx)
7
7
 
8
- An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. It also now provides detection and removal of personal information. Now includes hate speech and offensive language detection using AI.
8
+ An open-source Python library for data cleaning tasks. It includes functions for profanity detection, and removal, and detection and removal of personal information. Also includes hate speech and offensive language detection and removal, using AI.
9
9
 
10
10
  > [!IMPORTANT]
11
11
  > Please downgrade to `numpy` version `1.26.4`. Our ValX **DecisionTreeClassifier** AI model, relies on lower versions of `numpy`, because it was trained on these versions.
12
12
  > For more information see: https://techoverflow.net/2024/07/23/how-to-fix-numpy-dtype-size-changed-may-indicate-binary-incompatibility-expected-96-from-c-header-got-88-from-pyobject/
13
13
 
14
+ ## Changes in 0.2.4
15
+
16
+ Fixed a major incompatibility issue with `scikit-learn` due to version changes in `scikit-learn v1.3.0` which causes compatibility issues with versions later than `1.2.2`. ValX can now be used with `scikit-learn` versions earlier and later than `1.3.0`!
17
+
18
+ We've also removed `scikit-learn==1.2.2` as a dependency, as most versions of `scikit-learn` will now work.
19
+
20
+ ## Changes in 0.2.3
21
+
22
+ We have introduced a new optional `info_type` parameter into our `detect_sensitive_information`, and `remove_sensitive_information` functions, to allow you to have fine-grained control over what sensitive information you want to detect or remove.
23
+
24
+ Also introduced more detection patterns for other types of sensitive information, including:
25
+ - `"iban"`: International Bank Account Number.
26
+ - `"mrn"`: Medical Record Number (may not work correctly, depending on provider and country).
27
+ - `"icd10"`: International Classification of Diseases, Tenth Revision.
28
+ - `"geo_coords"`: Geo-coordinates (latitude and longitude in decimal degrees format).
29
+ - `"username"`: Username handles (@username).
30
+ - `"file_path"`: File paths (general patterns for both Windows and Unix paths).
31
+ - `"bitcoin_wallet"`: Cryptocurrency wallet address.
32
+ - `"ethereum_wallet"`: Cryptocurrency wallet addresses.
33
+
14
34
  ## Changes in 0.2.2
15
35
 
16
36
  We have refactored and changed the `detect_profanity` function:
@@ -116,6 +136,9 @@ from valx import detect_sensitive_information
116
136
  detected_sensitive_info = detect_sensitive_information(sample_text)
117
137
  ```
118
138
 
139
+ > [!NOTE]
140
+ > We have updated this function, and it now includes an optional argument for `info_type`, which can be used to detect only specific types of sensitive information. It was also added to `remove_sensitive_information`.
141
+
119
142
  ### Remove Sensitive Information
120
143
 
121
144
  ```python
@@ -5,7 +5,7 @@ with open("README.md", "r", encoding="utf-8") as fh:
5
5
 
6
6
  setup(
7
7
  name='valx',
8
- version='0.2.2',
8
+ version='0.2.4',
9
9
  author='Infinitode Pty Ltd',
10
10
  author_email='infinitode.ltd@gmail.com',
11
11
  description='An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.',
@@ -15,10 +15,10 @@ setup(
15
15
  packages=find_packages(),
16
16
  package_data={'valx': ['models/*']},
17
17
  install_requires=[
18
- 'scikit-learn==1.2.2' # for the AI to function properly
18
+ 'scikit-learn'
19
19
  ],
20
20
  classifiers=[
21
- 'Development Status :: 3 - Alpha',
21
+ 'Development Status :: 5 - Production/Stable',
22
22
  'Intended Audience :: Developers',
23
23
  'License :: OSI Approved :: MIT License',
24
24
  'Programming Language :: Python :: 3.6',
@@ -3,6 +3,7 @@ import re
3
3
 
4
4
  # New version, that includes AI detection
5
5
  import pickle
6
+ from sklearn import __version__ as sklearnversion
6
7
  from sklearn.tree import DecisionTreeClassifier # Import the DecisionTreeClassifier class
7
8
  from sklearn.feature_extraction.text import CountVectorizer # Import the CountVectorizer class
8
9
 
@@ -2825,85 +2826,68 @@ def remove_profanity(text_data, output_file=None, language='English'):
2825
2826
 
2826
2827
  return cleaned_data
2827
2828
 
2828
- # New functions for 1.5
2829
- def detect_sensitive_information(text_data):
2829
+ def detect_sensitive_information(text_data, info_type=["email", "phone", "credit_card", "ssn", "id", "address", "ip", "iban", "mrn", "icd10", "geo_coords", "username", "file_path", "bitcoin_wallet", "ethereum_wallet"]):
2830
2830
  """
2831
2831
  Detect sensitive information patterns in the provided text data.
2832
2832
 
2833
2833
  Args:
2834
2834
  text_data (list of str): A list of strings representing the text data to be analyzed.
2835
+ info_type (str or list of str, optional): One or more types of sensitive info to detect. Available types are: "email", "phone", "credit_card", "ssn", "id", "address", "ip", "iban", "mrn", "icd10", "geo_coords", "username", "file_path", "bitcoin_wallet", "ethereum_wallet". Uses all info types by default.
2835
2836
 
2836
2837
  Returns:
2837
2838
  list of tuple: A list of tuples containing detected sensitive information, each tuple
2838
2839
  representing (line number, column index, type, value).
2839
2840
  """
2841
+ if isinstance(info_type, str):
2842
+ info_type = [info_type] # Convert single string to list for consistent handling
2843
+
2844
+ # Dictionary to map info_type to regex patterns
2845
+ regex_patterns = {
2846
+ "email": r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b',
2847
+ "phone": r'\b(?:\d{3}[-.\s]??\d{3}[-.\s]??\d{4}|\(\d{3}\)\s*\d{3}[-.\s]??\d{4}|\d{3}[-.\s]??\d{4})\b',
2848
+ "credit_card": r'\b(?:\d[ -]*?){13,16}\b',
2849
+ "ssn": r'\b\d{3}[-]?\d{2}[-]?\d{4}\b',
2850
+ "id": r'\b[A-Za-z]{1,2}\d{6,9}\b',
2851
+ "address": r'\b\d+\s\w+\s\w+,\s\w+,\s\w+\b',
2852
+ "ip": r'\b(?:[0-9]{1,3}\.){3}[0-9]{1,3}\b',
2853
+ "iban": r'\b[A-Z]{2}[0-9]{2}[A-Z0-9]{1,30}\b',
2854
+ "mrn": r'\b\d{5,10}\b',
2855
+ "icd10": r'\b[A-TV-Z][0-9]{2}(?:\.[0-9A-TV-Z]{1,4})?\b',
2856
+ "geo_coords": r'\b[-+]?([1-8]?\d(\.\d+)?|90(\.0+)?),\s*[-+]?(180(\.0+)?|((1[0-7]\d)|([1-9]?\d))(\.\d+)?)\b',
2857
+ "username": r'@\w+',
2858
+ "file_path": r'([A-Za-z]:\\|/)?([-\w\s]+\\|/)*[-\w\s]+\.\w+',
2859
+ "bitcoin_wallet": r'\b[13][a-km-zA-HJ-NP-Z1-9]{25,34}\b',
2860
+ "ethereum_wallet": r'\b0x[a-fA-F0-9]{40}\b'
2861
+ }
2862
+
2840
2863
  sensitive_info = []
2841
2864
 
2842
- # Regular expression for detecting email addresses
2843
- email_regex = r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b'
2844
- # Regular expression for detecting phone numbers
2845
- phone_regex = r'\b(?:\d{3}[-.\s]??\d{3}[-.\s]??\d{4}|\(\d{3}\)\s*\d{3}[-.\s]??\d{4}|\d{3}[-.\s]??\d{4})\b'
2846
- # Regular expression for detecting credit card numbers (Visa, Mastercard, American Express, Discover)
2847
- credit_card_regex = r'\b(?:\d[ -]*?){13,16}\b'
2848
- # Regular expression for detecting social security numbers (SSN)
2849
- ssn_regex = r'\b\d{3}[-]?\d{2}[-]?\d{4}\b'
2850
- # Regular expression for detecting ID numbers (e.g., driver's license, passport)
2851
- id_number_regex = r'\b[A-Za-z]{1,2}\d{6,9}\b'
2852
- # Regular expression for detecting addresses (simple example)
2853
- address_regex = r'\b\d+\s\w+\s\w+,\s\w+,\s\w+\b'
2854
- # Regular expression for detecting IP addresses
2855
- ip_address_regex = r'\b(?:[0-9]{1,3}\.){3}[0-9]{1,3}\b'
2856
-
2865
+ # Loop through each line and check for sensitive info types based on specified `info_type`
2857
2866
  for i, line in enumerate(text_data):
2858
- # Detect email addresses
2859
- emails = re.findall(email_regex, line)
2860
- for email in emails:
2861
- sensitive_info.append((i + 1, line.find(email), 'Email', email))
2862
-
2863
- # Detect phone numbers
2864
- phones = re.findall(phone_regex, line)
2865
- for phone in phones:
2866
- sensitive_info.append((i + 1, line.find(phone), 'Phone', phone))
2867
-
2868
- # Detect credit card numbers
2869
- credit_cards = re.findall(credit_card_regex, line)
2870
- for card in credit_cards:
2871
- sensitive_info.append((i + 1, line.find(card), 'Credit Card', card))
2872
-
2873
- # Detect social security numbers
2874
- ssn_numbers = re.findall(ssn_regex, line)
2875
- for ssn in ssn_numbers:
2876
- sensitive_info.append((i + 1, line.find(ssn), 'SSN', ssn))
2877
-
2878
- # Detect ID numbers
2879
- id_numbers = re.findall(id_number_regex, line)
2880
- for id_number in id_numbers:
2881
- sensitive_info.append((i + 1, line.find(id_number), 'ID Number', id_number))
2882
-
2883
- # Detect addresses
2884
- addresses = re.findall(address_regex, line)
2885
- for address in addresses:
2886
- sensitive_info.append((i + 1, line.find(address), 'Address', address))
2887
-
2888
- # Detect IP addresses
2889
- ip_addresses = re.findall(ip_address_regex, line)
2890
- for ip_address in ip_addresses:
2891
- sensitive_info.append((i + 1, line.find(ip_address), 'IP Address', ip_address))
2892
-
2867
+ for type_ in info_type:
2868
+ pattern = regex_patterns.get(type_)
2869
+ if pattern:
2870
+ matches = re.finditer(pattern, line)
2871
+ for match in matches:
2872
+ sensitive_info.append((i + 1, match.start(), type_.capitalize(), match.group()))
2873
+
2893
2874
  return sensitive_info
2894
2875
 
2895
- def remove_sensitive_information(text_data, output_file=None):
2876
+ def remove_sensitive_information(text_data, output_file=None, info_type=["email", "phone", "credit_card", "ssn", "id", "address", "ip", "iban", "mrn", "icd10", "geo_coords", "username", "file_path", "bitcoin_wallet", "ethereum_wallet"]):
2896
2877
  """
2897
2878
  Remove sensitive information patterns from the provided text data.
2898
2879
 
2899
2880
  Args:
2900
2881
  text_data (list of str): A list of strings representing the text data to be cleaned.
2901
2882
  output_file (str, optional): Path to the output file where cleaned data will be saved.
2883
+ info_type (str or list of str, optional): One or more types of sensitive info to detect and remove. Available types are: "email", "phone", "credit_card", "ssn", "id", "address", "ip", "iban", "mrn", "icd10", "geo_coords", "username", "file_path", "bitcoin_wallet", "ethereum_wallet". Uses all info types by default.
2902
2884
 
2903
2885
  Returns:
2904
2886
  list of str: A list of strings representing the cleaned text data.
2905
2887
  """
2906
- sensitive_info = detect_sensitive_information(text_data)
2888
+ # Detect sensitive information
2889
+ sensitive_info = detect_sensitive_information(text_data, info_type=info_type)
2890
+
2907
2891
  cleaned_data = []
2908
2892
  for line in text_data:
2909
2893
  cleaned_line = line
@@ -2917,11 +2901,20 @@ def remove_sensitive_information(text_data, output_file=None):
2917
2901
  file.write('\n'.join(cleaned_data))
2918
2902
  return cleaned_data
2919
2903
 
2920
-
2921
2904
  # New version
2922
2905
  MODEL_DIR = os.path.join(os.path.dirname(__file__), 'models')
2923
- DECISION_TREE_MODEL_PATH = os.path.join(MODEL_DIR, 'decision_tree_model.sav')
2924
2906
  COUNT_VECTORIZER_PATH = os.path.join(MODEL_DIR, 'count_vectorizer.sav')
2907
+ DEFAULT_MODEL_FILENAME = "decision_tree_model.sav"
2908
+ UPGRADED_MODEL_FILENAME = "classifier_upgraded.pkl"
2909
+
2910
+ # Check scikit-learn version
2911
+ SKLEARN_VERSION = tuple(map(int, sklearnversion.split(".")))
2912
+
2913
+ # Set the model path based on the version
2914
+ if SKLEARN_VERSION >= (1, 3, 0):
2915
+ DECISION_TREE_MODEL_PATH = os.path.join(MODEL_DIR, UPGRADED_MODEL_FILENAME)
2916
+ else:
2917
+ DECISION_TREE_MODEL_PATH = os.path.join(MODEL_DIR, DEFAULT_MODEL_FILENAME)
2925
2918
 
2926
2919
  # Load the saved models
2927
2920
  model = pickle.load(open(DECISION_TREE_MODEL_PATH, 'rb'))
@@ -1,11 +1,11 @@
1
1
  Metadata-Version: 2.1
2
2
  Name: valx
3
- Version: 0.2.2
3
+ Version: 0.2.4
4
4
  Summary: An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.
5
5
  Home-page: https://github.com/infinitode/valx
6
6
  Author: Infinitode Pty Ltd
7
7
  Author-email: infinitode.ltd@gmail.com
8
- Classifier: Development Status :: 3 - Alpha
8
+ Classifier: Development Status :: 5 - Production/Stable
9
9
  Classifier: Intended Audience :: Developers
10
10
  Classifier: License :: OSI Approved :: MIT License
11
11
  Classifier: Programming Language :: Python :: 3.6
@@ -18,7 +18,7 @@ Classifier: Programming Language :: Python :: 3.12
18
18
  Requires-Python: >=3.6
19
19
  Description-Content-Type: text/markdown
20
20
  License-File: LICENSE
21
- Requires-Dist: scikit-learn==1.2.2
21
+ Requires-Dist: scikit-learn
22
22
 
23
23
  # ValX
24
24
  ![Python Version](https://img.shields.io/badge/python-3.12-blue.svg)
@@ -27,12 +27,32 @@ Requires-Dist: scikit-learn==1.2.2
27
27
  ![License Compliance](https://img.shields.io/badge/license-compliance-brightgreen.svg)
28
28
  ![PyPI Version](https://img.shields.io/pypi/v/valx)
29
29
 
30
- An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. It also now provides detection and removal of personal information. Now includes hate speech and offensive language detection using AI.
30
+ An open-source Python library for data cleaning tasks. It includes functions for profanity detection, and removal, and detection and removal of personal information. Also includes hate speech and offensive language detection and removal, using AI.
31
31
 
32
32
  > [!IMPORTANT]
33
33
  > Please downgrade to `numpy` version `1.26.4`. Our ValX **DecisionTreeClassifier** AI model, relies on lower versions of `numpy`, because it was trained on these versions.
34
34
  > For more information see: https://techoverflow.net/2024/07/23/how-to-fix-numpy-dtype-size-changed-may-indicate-binary-incompatibility-expected-96-from-c-header-got-88-from-pyobject/
35
35
 
36
+ ## Changes in 0.2.4
37
+
38
+ Fixed a major incompatibility issue with `scikit-learn` due to version changes in `scikit-learn v1.3.0` which causes compatibility issues with versions later than `1.2.2`. ValX can now be used with `scikit-learn` versions earlier and later than `1.3.0`!
39
+
40
+ We've also removed `scikit-learn==1.2.2` as a dependency, as most versions of `scikit-learn` will now work.
41
+
42
+ ## Changes in 0.2.3
43
+
44
+ We have introduced a new optional `info_type` parameter into our `detect_sensitive_information`, and `remove_sensitive_information` functions, to allow you to have fine-grained control over what sensitive information you want to detect or remove.
45
+
46
+ Also introduced more detection patterns for other types of sensitive information, including:
47
+ - `"iban"`: International Bank Account Number.
48
+ - `"mrn"`: Medical Record Number (may not work correctly, depending on provider and country).
49
+ - `"icd10"`: International Classification of Diseases, Tenth Revision.
50
+ - `"geo_coords"`: Geo-coordinates (latitude and longitude in decimal degrees format).
51
+ - `"username"`: Username handles (@username).
52
+ - `"file_path"`: File paths (general patterns for both Windows and Unix paths).
53
+ - `"bitcoin_wallet"`: Cryptocurrency wallet address.
54
+ - `"ethereum_wallet"`: Cryptocurrency wallet addresses.
55
+
36
56
  ## Changes in 0.2.2
37
57
 
38
58
  We have refactored and changed the `detect_profanity` function:
@@ -138,6 +158,9 @@ from valx import detect_sensitive_information
138
158
  detected_sensitive_info = detect_sensitive_information(sample_text)
139
159
  ```
140
160
 
161
+ > [!NOTE]
162
+ > We have updated this function, and it now includes an optional argument for `info_type`, which can be used to detect only specific types of sensitive information. It was also added to `remove_sensitive_information`.
163
+
141
164
  ### Remove Sensitive Information
142
165
 
143
166
  ```python
@@ -7,5 +7,6 @@ valx.egg-info/SOURCES.txt
7
7
  valx.egg-info/dependency_links.txt
8
8
  valx.egg-info/requires.txt
9
9
  valx.egg-info/top_level.txt
10
+ valx/models/classifier_upgraded.pkl
10
11
  valx/models/count_vectorizer.sav
11
12
  valx/models/decision_tree_model.sav
@@ -0,0 +1 @@
1
+ scikit-learn
@@ -1 +0,0 @@
1
- scikit-learn==1.2.2
File without changes
File without changes
File without changes