valx 0.2.2__tar.gz → 0.2.3__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.1
2
2
  Name: valx
3
- Version: 0.2.2
3
+ Version: 0.2.3
4
4
  Summary: An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.
5
5
  Home-page: https://github.com/infinitode/valx
6
6
  Author: Infinitode Pty Ltd
@@ -27,12 +27,26 @@ Requires-Dist: scikit-learn==1.2.2
27
27
  ![License Compliance](https://img.shields.io/badge/license-compliance-brightgreen.svg)
28
28
  ![PyPI Version](https://img.shields.io/pypi/v/valx)
29
29
 
30
- An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. It also now provides detection and removal of personal information. Now includes hate speech and offensive language detection using AI.
30
+ An open-source Python library for data cleaning tasks. It includes functions for profanity detection, and removal, and detection and removal of personal information. Also includes hate speech and offensive language detection and removal, using AI.
31
31
 
32
32
  > [!IMPORTANT]
33
33
  > Please downgrade to `numpy` version `1.26.4`. Our ValX **DecisionTreeClassifier** AI model, relies on lower versions of `numpy`, because it was trained on these versions.
34
34
  > For more information see: https://techoverflow.net/2024/07/23/how-to-fix-numpy-dtype-size-changed-may-indicate-binary-incompatibility-expected-96-from-c-header-got-88-from-pyobject/
35
35
 
36
+ ## Changes in 0.2.3
37
+
38
+ We have introduced a new optional `info_type` parameter into our `detect_sensitive_information`, and `remove_sensitive_information` functions, to allow you to have fine-grained control over what sensitive information you want to detect or remove.
39
+
40
+ Also introduced more detection patterns for other types of sensitive information, including:
41
+ - `"iban"`: International Bank Account Number.
42
+ - `"mrn"`: Medical Record Number (may not work correctly, depending on provider and country).
43
+ - `"icd10"`: International Classification of Diseases, Tenth Revision.
44
+ - `"geo_coords"`: Geo-coordinates (latitude and longitude in decimal degrees format).
45
+ - `"username"`: Username handles (@username).
46
+ - `"file_path"`: File paths (general patterns for both Windows and Unix paths).
47
+ - `"bitcoin_wallet"`: Cryptocurrency wallet address.
48
+ - `"ethereum_wallet"`: Cryptocurrency wallet addresses.
49
+
36
50
  ## Changes in 0.2.2
37
51
 
38
52
  We have refactored and changed the `detect_profanity` function:
@@ -138,6 +152,9 @@ from valx import detect_sensitive_information
138
152
  detected_sensitive_info = detect_sensitive_information(sample_text)
139
153
  ```
140
154
 
155
+ > [!NOTE]
156
+ > We have updated this function, and it now includes an optional argument for `info_type`, which can be used to detect only specific types of sensitive information. It was also added to `remove_sensitive_information`.
157
+
141
158
  ### Remove Sensitive Information
142
159
 
143
160
  ```python
@@ -5,12 +5,26 @@
5
5
  ![License Compliance](https://img.shields.io/badge/license-compliance-brightgreen.svg)
6
6
  ![PyPI Version](https://img.shields.io/pypi/v/valx)
7
7
 
8
- An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. It also now provides detection and removal of personal information. Now includes hate speech and offensive language detection using AI.
8
+ An open-source Python library for data cleaning tasks. It includes functions for profanity detection, and removal, and detection and removal of personal information. Also includes hate speech and offensive language detection and removal, using AI.
9
9
 
10
10
  > [!IMPORTANT]
11
11
  > Please downgrade to `numpy` version `1.26.4`. Our ValX **DecisionTreeClassifier** AI model, relies on lower versions of `numpy`, because it was trained on these versions.
12
12
  > For more information see: https://techoverflow.net/2024/07/23/how-to-fix-numpy-dtype-size-changed-may-indicate-binary-incompatibility-expected-96-from-c-header-got-88-from-pyobject/
13
13
 
14
+ ## Changes in 0.2.3
15
+
16
+ We have introduced a new optional `info_type` parameter into our `detect_sensitive_information`, and `remove_sensitive_information` functions, to allow you to have fine-grained control over what sensitive information you want to detect or remove.
17
+
18
+ Also introduced more detection patterns for other types of sensitive information, including:
19
+ - `"iban"`: International Bank Account Number.
20
+ - `"mrn"`: Medical Record Number (may not work correctly, depending on provider and country).
21
+ - `"icd10"`: International Classification of Diseases, Tenth Revision.
22
+ - `"geo_coords"`: Geo-coordinates (latitude and longitude in decimal degrees format).
23
+ - `"username"`: Username handles (@username).
24
+ - `"file_path"`: File paths (general patterns for both Windows and Unix paths).
25
+ - `"bitcoin_wallet"`: Cryptocurrency wallet address.
26
+ - `"ethereum_wallet"`: Cryptocurrency wallet addresses.
27
+
14
28
  ## Changes in 0.2.2
15
29
 
16
30
  We have refactored and changed the `detect_profanity` function:
@@ -116,6 +130,9 @@ from valx import detect_sensitive_information
116
130
  detected_sensitive_info = detect_sensitive_information(sample_text)
117
131
  ```
118
132
 
133
+ > [!NOTE]
134
+ > We have updated this function, and it now includes an optional argument for `info_type`, which can be used to detect only specific types of sensitive information. It was also added to `remove_sensitive_information`.
135
+
119
136
  ### Remove Sensitive Information
120
137
 
121
138
  ```python
@@ -5,7 +5,7 @@ with open("README.md", "r", encoding="utf-8") as fh:
5
5
 
6
6
  setup(
7
7
  name='valx',
8
- version='0.2.2',
8
+ version='0.2.3',
9
9
  author='Infinitode Pty Ltd',
10
10
  author_email='infinitode.ltd@gmail.com',
11
11
  description='An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.',
@@ -2825,85 +2825,68 @@ def remove_profanity(text_data, output_file=None, language='English'):
2825
2825
 
2826
2826
  return cleaned_data
2827
2827
 
2828
- # New functions for 1.5
2829
- def detect_sensitive_information(text_data):
2828
+ def detect_sensitive_information(text_data, info_type=["email", "phone", "credit_card", "ssn", "id", "address", "ip", "iban", "mrn", "icd10", "geo_coords", "username", "file_path", "bitcoin_wallet", "ethereum_wallet"]):
2830
2829
  """
2831
2830
  Detect sensitive information patterns in the provided text data.
2832
2831
 
2833
2832
  Args:
2834
2833
  text_data (list of str): A list of strings representing the text data to be analyzed.
2834
+ info_type (str or list of str, optional): One or more types of sensitive info to detect. Available types are: "email", "phone", "credit_card", "ssn", "id", "address", "ip", "iban", "mrn", "icd10", "geo_coords", "username", "file_path", "bitcoin_wallet", "ethereum_wallet". Uses all info types by default.
2835
2835
 
2836
2836
  Returns:
2837
2837
  list of tuple: A list of tuples containing detected sensitive information, each tuple
2838
2838
  representing (line number, column index, type, value).
2839
2839
  """
2840
+ if isinstance(info_type, str):
2841
+ info_type = [info_type] # Convert single string to list for consistent handling
2842
+
2843
+ # Dictionary to map info_type to regex patterns
2844
+ regex_patterns = {
2845
+ "email": r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b',
2846
+ "phone": r'\b(?:\d{3}[-.\s]??\d{3}[-.\s]??\d{4}|\(\d{3}\)\s*\d{3}[-.\s]??\d{4}|\d{3}[-.\s]??\d{4})\b',
2847
+ "credit_card": r'\b(?:\d[ -]*?){13,16}\b',
2848
+ "ssn": r'\b\d{3}[-]?\d{2}[-]?\d{4}\b',
2849
+ "id": r'\b[A-Za-z]{1,2}\d{6,9}\b',
2850
+ "address": r'\b\d+\s\w+\s\w+,\s\w+,\s\w+\b',
2851
+ "ip": r'\b(?:[0-9]{1,3}\.){3}[0-9]{1,3}\b',
2852
+ "iban": r'\b[A-Z]{2}[0-9]{2}[A-Z0-9]{1,30}\b',
2853
+ "mrn": r'\b\d{5,10}\b',
2854
+ "icd10": r'\b[A-TV-Z][0-9]{2}(?:\.[0-9A-TV-Z]{1,4})?\b',
2855
+ "geo_coords": r'\b[-+]?([1-8]?\d(\.\d+)?|90(\.0+)?),\s*[-+]?(180(\.0+)?|((1[0-7]\d)|([1-9]?\d))(\.\d+)?)\b',
2856
+ "username": r'@\w+',
2857
+ "file_path": r'([A-Za-z]:\\|/)?([-\w\s]+\\|/)*[-\w\s]+\.\w+',
2858
+ "bitcoin_wallet": r'\b[13][a-km-zA-HJ-NP-Z1-9]{25,34}\b',
2859
+ "ethereum_wallet": r'\b0x[a-fA-F0-9]{40}\b'
2860
+ }
2861
+
2840
2862
  sensitive_info = []
2841
2863
 
2842
- # Regular expression for detecting email addresses
2843
- email_regex = r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b'
2844
- # Regular expression for detecting phone numbers
2845
- phone_regex = r'\b(?:\d{3}[-.\s]??\d{3}[-.\s]??\d{4}|\(\d{3}\)\s*\d{3}[-.\s]??\d{4}|\d{3}[-.\s]??\d{4})\b'
2846
- # Regular expression for detecting credit card numbers (Visa, Mastercard, American Express, Discover)
2847
- credit_card_regex = r'\b(?:\d[ -]*?){13,16}\b'
2848
- # Regular expression for detecting social security numbers (SSN)
2849
- ssn_regex = r'\b\d{3}[-]?\d{2}[-]?\d{4}\b'
2850
- # Regular expression for detecting ID numbers (e.g., driver's license, passport)
2851
- id_number_regex = r'\b[A-Za-z]{1,2}\d{6,9}\b'
2852
- # Regular expression for detecting addresses (simple example)
2853
- address_regex = r'\b\d+\s\w+\s\w+,\s\w+,\s\w+\b'
2854
- # Regular expression for detecting IP addresses
2855
- ip_address_regex = r'\b(?:[0-9]{1,3}\.){3}[0-9]{1,3}\b'
2856
-
2864
+ # Loop through each line and check for sensitive info types based on specified `info_type`
2857
2865
  for i, line in enumerate(text_data):
2858
- # Detect email addresses
2859
- emails = re.findall(email_regex, line)
2860
- for email in emails:
2861
- sensitive_info.append((i + 1, line.find(email), 'Email', email))
2862
-
2863
- # Detect phone numbers
2864
- phones = re.findall(phone_regex, line)
2865
- for phone in phones:
2866
- sensitive_info.append((i + 1, line.find(phone), 'Phone', phone))
2867
-
2868
- # Detect credit card numbers
2869
- credit_cards = re.findall(credit_card_regex, line)
2870
- for card in credit_cards:
2871
- sensitive_info.append((i + 1, line.find(card), 'Credit Card', card))
2872
-
2873
- # Detect social security numbers
2874
- ssn_numbers = re.findall(ssn_regex, line)
2875
- for ssn in ssn_numbers:
2876
- sensitive_info.append((i + 1, line.find(ssn), 'SSN', ssn))
2877
-
2878
- # Detect ID numbers
2879
- id_numbers = re.findall(id_number_regex, line)
2880
- for id_number in id_numbers:
2881
- sensitive_info.append((i + 1, line.find(id_number), 'ID Number', id_number))
2882
-
2883
- # Detect addresses
2884
- addresses = re.findall(address_regex, line)
2885
- for address in addresses:
2886
- sensitive_info.append((i + 1, line.find(address), 'Address', address))
2887
-
2888
- # Detect IP addresses
2889
- ip_addresses = re.findall(ip_address_regex, line)
2890
- for ip_address in ip_addresses:
2891
- sensitive_info.append((i + 1, line.find(ip_address), 'IP Address', ip_address))
2892
-
2866
+ for type_ in info_type:
2867
+ pattern = regex_patterns.get(type_)
2868
+ if pattern:
2869
+ matches = re.finditer(pattern, line)
2870
+ for match in matches:
2871
+ sensitive_info.append((i + 1, match.start(), type_.capitalize(), match.group()))
2872
+
2893
2873
  return sensitive_info
2894
2874
 
2895
- def remove_sensitive_information(text_data, output_file=None):
2875
+ def remove_sensitive_information(text_data, output_file=None, info_type=["email", "phone", "credit_card", "ssn", "id", "address", "ip", "iban", "mrn", "icd10", "geo_coords", "username", "file_path", "bitcoin_wallet", "ethereum_wallet"]):
2896
2876
  """
2897
2877
  Remove sensitive information patterns from the provided text data.
2898
2878
 
2899
2879
  Args:
2900
2880
  text_data (list of str): A list of strings representing the text data to be cleaned.
2901
2881
  output_file (str, optional): Path to the output file where cleaned data will be saved.
2882
+ info_type (str or list of str, optional): One or more types of sensitive info to detect and remove. Available types are: "email", "phone", "credit_card", "ssn", "id", "address", "ip", "iban", "mrn", "icd10", "geo_coords", "username", "file_path", "bitcoin_wallet", "ethereum_wallet". Uses all info types by default.
2902
2883
 
2903
2884
  Returns:
2904
2885
  list of str: A list of strings representing the cleaned text data.
2905
2886
  """
2906
- sensitive_info = detect_sensitive_information(text_data)
2887
+ # Detect sensitive information
2888
+ sensitive_info = detect_sensitive_information(text_data, info_type=info_type)
2889
+
2907
2890
  cleaned_data = []
2908
2891
  for line in text_data:
2909
2892
  cleaned_line = line
@@ -2917,7 +2900,6 @@ def remove_sensitive_information(text_data, output_file=None):
2917
2900
  file.write('\n'.join(cleaned_data))
2918
2901
  return cleaned_data
2919
2902
 
2920
-
2921
2903
  # New version
2922
2904
  MODEL_DIR = os.path.join(os.path.dirname(__file__), 'models')
2923
2905
  DECISION_TREE_MODEL_PATH = os.path.join(MODEL_DIR, 'decision_tree_model.sav')
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.1
2
2
  Name: valx
3
- Version: 0.2.2
3
+ Version: 0.2.3
4
4
  Summary: An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.
5
5
  Home-page: https://github.com/infinitode/valx
6
6
  Author: Infinitode Pty Ltd
@@ -27,12 +27,26 @@ Requires-Dist: scikit-learn==1.2.2
27
27
  ![License Compliance](https://img.shields.io/badge/license-compliance-brightgreen.svg)
28
28
  ![PyPI Version](https://img.shields.io/pypi/v/valx)
29
29
 
30
- An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. It also now provides detection and removal of personal information. Now includes hate speech and offensive language detection using AI.
30
+ An open-source Python library for data cleaning tasks. It includes functions for profanity detection, and removal, and detection and removal of personal information. Also includes hate speech and offensive language detection and removal, using AI.
31
31
 
32
32
  > [!IMPORTANT]
33
33
  > Please downgrade to `numpy` version `1.26.4`. Our ValX **DecisionTreeClassifier** AI model, relies on lower versions of `numpy`, because it was trained on these versions.
34
34
  > For more information see: https://techoverflow.net/2024/07/23/how-to-fix-numpy-dtype-size-changed-may-indicate-binary-incompatibility-expected-96-from-c-header-got-88-from-pyobject/
35
35
 
36
+ ## Changes in 0.2.3
37
+
38
+ We have introduced a new optional `info_type` parameter into our `detect_sensitive_information`, and `remove_sensitive_information` functions, to allow you to have fine-grained control over what sensitive information you want to detect or remove.
39
+
40
+ Also introduced more detection patterns for other types of sensitive information, including:
41
+ - `"iban"`: International Bank Account Number.
42
+ - `"mrn"`: Medical Record Number (may not work correctly, depending on provider and country).
43
+ - `"icd10"`: International Classification of Diseases, Tenth Revision.
44
+ - `"geo_coords"`: Geo-coordinates (latitude and longitude in decimal degrees format).
45
+ - `"username"`: Username handles (@username).
46
+ - `"file_path"`: File paths (general patterns for both Windows and Unix paths).
47
+ - `"bitcoin_wallet"`: Cryptocurrency wallet address.
48
+ - `"ethereum_wallet"`: Cryptocurrency wallet addresses.
49
+
36
50
  ## Changes in 0.2.2
37
51
 
38
52
  We have refactored and changed the `detect_profanity` function:
@@ -138,6 +152,9 @@ from valx import detect_sensitive_information
138
152
  detected_sensitive_info = detect_sensitive_information(sample_text)
139
153
  ```
140
154
 
155
+ > [!NOTE]
156
+ > We have updated this function, and it now includes an optional argument for `info_type`, which can be used to detect only specific types of sensitive information. It was also added to `remove_sensitive_information`.
157
+
141
158
  ### Remove Sensitive Information
142
159
 
143
160
  ```python
File without changes
File without changes
File without changes
File without changes
File without changes