valx 0.2.1__tar.gz → 0.2.3__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.1
2
2
  Name: valx
3
- Version: 0.2.1
3
+ Version: 0.2.3
4
4
  Summary: An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.
5
5
  Home-page: https://github.com/infinitode/valx
6
6
  Author: Infinitode Pty Ltd
@@ -27,7 +27,34 @@ Requires-Dist: scikit-learn==1.2.2
27
27
  ![License Compliance](https://img.shields.io/badge/license-compliance-brightgreen.svg)
28
28
  ![PyPI Version](https://img.shields.io/pypi/v/valx)
29
29
 
30
- An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Also now includes personal information detection and removal. Now includes hate speech and offensive language detection using AI.
30
+ An open-source Python library for data cleaning tasks. It includes functions for profanity detection, and removal, and detection and removal of personal information. Also includes hate speech and offensive language detection and removal, using AI.
31
+
32
+ > [!IMPORTANT]
33
+ > Please downgrade to `numpy` version `1.26.4`. Our ValX **DecisionTreeClassifier** AI model, relies on lower versions of `numpy`, because it was trained on these versions.
34
+ > For more information see: https://techoverflow.net/2024/07/23/how-to-fix-numpy-dtype-size-changed-may-indicate-binary-incompatibility-expected-96-from-c-header-got-88-from-pyobject/
35
+
36
+ ## Changes in 0.2.3
37
+
38
+ We have introduced a new optional `info_type` parameter into our `detect_sensitive_information`, and `remove_sensitive_information` functions, to allow you to have fine-grained control over what sensitive information you want to detect or remove.
39
+
40
+ Also introduced more detection patterns for other types of sensitive information, including:
41
+ - `"iban"`: International Bank Account Number.
42
+ - `"mrn"`: Medical Record Number (may not work correctly, depending on provider and country).
43
+ - `"icd10"`: International Classification of Diseases, Tenth Revision.
44
+ - `"geo_coords"`: Geo-coordinates (latitude and longitude in decimal degrees format).
45
+ - `"username"`: Username handles (@username).
46
+ - `"file_path"`: File paths (general patterns for both Windows and Unix paths).
47
+ - `"bitcoin_wallet"`: Cryptocurrency wallet address.
48
+ - `"ethereum_wallet"`: Cryptocurrency wallet addresses.
49
+
50
+ ## Changes in 0.2.2
51
+
52
+ We have refactored and changed the `detect_profanity` function:
53
+ - Removed unnecessary printing
54
+ - Now returns more information about each found profanity, including `Line`, `Column`, `Word`, and `Language`.
55
+
56
+ > [!NOTE]
57
+ > You can view [ValX's package documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more information on changes.
31
58
 
32
59
  ## Changes in 0.2.1
33
60
 
@@ -63,6 +90,38 @@ Please ensure that you have one of these Python versions installed before using
63
90
  - **Detect Hate Speech**: Detect hate speech or offensive speech in text, using AI.
64
91
  - **Remove Hate Speech**: Remove hate speech or offensive speech in text, using AI.
65
92
 
93
+ ### List of supported languages for profanity detection and removal
94
+ Below is a complete list of all the available supported languages for ValX's profanity detection and removal functions which are valid values for `language`:
95
+
96
+ - **All**
97
+ - Arabic
98
+ - Czech
99
+ - Danish
100
+ - German
101
+ - English
102
+ - Esperanto
103
+ - Persian
104
+ - Finnish
105
+ - Filipino
106
+ - French
107
+ - French (CA)
108
+ - Hindi
109
+ - Hungarian
110
+ - Italian
111
+ - Japanese
112
+ - Kabyle
113
+ - Korean
114
+ - Dutch
115
+ - Norwegian
116
+ - Polish
117
+ - Portuguese
118
+ - Russian
119
+ - Swedish
120
+ - Thai
121
+ - Klingon
122
+ - Turkish
123
+ - Chinese
124
+
66
125
  ## Usage
67
126
 
68
127
  ### Detect Profanity
@@ -71,7 +130,8 @@ Please ensure that you have one of these Python versions installed before using
71
130
  from valx import detect_profanity
72
131
 
73
132
  # Detect profanity
74
- num_profanities = detect_profanity(sample_text, language='English')
133
+ results = detect_profanity(sample_text, language='English')
134
+ print("Profanity Evaluation Results", results)
75
135
  ```
76
136
 
77
137
  ### Remove Profanity
@@ -92,6 +152,9 @@ from valx import detect_sensitive_information
92
152
  detected_sensitive_info = detect_sensitive_information(sample_text)
93
153
  ```
94
154
 
155
+ > [!NOTE]
156
+ > We have updated this function, and it now includes an optional argument for `info_type`, which can be used to detect only specific types of sensitive information. It was also added to `remove_sensitive_information`.
157
+
95
158
  ### Remove Sensitive Information
96
159
 
97
160
  ```python
@@ -110,7 +173,14 @@ from valx import detect_hate_speech
110
173
  outcome_of_detection = detect_hate_speech("You are stupid.")
111
174
  ```
112
175
 
113
- > See our [official documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more examples on how to use ValX.*
176
+ > [!IMPORTANT]
177
+ > The model's possible outputs are:
178
+ > - `['Hate Speech']`: The text was flagged and contained hate speech.
179
+ > - `['Offensive Speech']`: The text was flagged and contained offensive speech.
180
+ > - `['No Hate and Offensive Speech']`: The text was not flagged for any hate speech or offensive speech.
181
+
182
+ > [!NOTE]
183
+ > See our [official documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more examples on how to use **ValX**.
114
184
 
115
185
  ## Contributing
116
186
 
@@ -5,7 +5,34 @@
5
5
  ![License Compliance](https://img.shields.io/badge/license-compliance-brightgreen.svg)
6
6
  ![PyPI Version](https://img.shields.io/pypi/v/valx)
7
7
 
8
- An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Also now includes personal information detection and removal. Now includes hate speech and offensive language detection using AI.
8
+ An open-source Python library for data cleaning tasks. It includes functions for profanity detection, and removal, and detection and removal of personal information. Also includes hate speech and offensive language detection and removal, using AI.
9
+
10
+ > [!IMPORTANT]
11
+ > Please downgrade to `numpy` version `1.26.4`. Our ValX **DecisionTreeClassifier** AI model, relies on lower versions of `numpy`, because it was trained on these versions.
12
+ > For more information see: https://techoverflow.net/2024/07/23/how-to-fix-numpy-dtype-size-changed-may-indicate-binary-incompatibility-expected-96-from-c-header-got-88-from-pyobject/
13
+
14
+ ## Changes in 0.2.3
15
+
16
+ We have introduced a new optional `info_type` parameter into our `detect_sensitive_information`, and `remove_sensitive_information` functions, to allow you to have fine-grained control over what sensitive information you want to detect or remove.
17
+
18
+ Also introduced more detection patterns for other types of sensitive information, including:
19
+ - `"iban"`: International Bank Account Number.
20
+ - `"mrn"`: Medical Record Number (may not work correctly, depending on provider and country).
21
+ - `"icd10"`: International Classification of Diseases, Tenth Revision.
22
+ - `"geo_coords"`: Geo-coordinates (latitude and longitude in decimal degrees format).
23
+ - `"username"`: Username handles (@username).
24
+ - `"file_path"`: File paths (general patterns for both Windows and Unix paths).
25
+ - `"bitcoin_wallet"`: Cryptocurrency wallet address.
26
+ - `"ethereum_wallet"`: Cryptocurrency wallet addresses.
27
+
28
+ ## Changes in 0.2.2
29
+
30
+ We have refactored and changed the `detect_profanity` function:
31
+ - Removed unnecessary printing
32
+ - Now returns more information about each found profanity, including `Line`, `Column`, `Word`, and `Language`.
33
+
34
+ > [!NOTE]
35
+ > You can view [ValX's package documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more information on changes.
9
36
 
10
37
  ## Changes in 0.2.1
11
38
 
@@ -41,6 +68,38 @@ Please ensure that you have one of these Python versions installed before using
41
68
  - **Detect Hate Speech**: Detect hate speech or offensive speech in text, using AI.
42
69
  - **Remove Hate Speech**: Remove hate speech or offensive speech in text, using AI.
43
70
 
71
+ ### List of supported languages for profanity detection and removal
72
+ Below is a complete list of all the available supported languages for ValX's profanity detection and removal functions which are valid values for `language`:
73
+
74
+ - **All**
75
+ - Arabic
76
+ - Czech
77
+ - Danish
78
+ - German
79
+ - English
80
+ - Esperanto
81
+ - Persian
82
+ - Finnish
83
+ - Filipino
84
+ - French
85
+ - French (CA)
86
+ - Hindi
87
+ - Hungarian
88
+ - Italian
89
+ - Japanese
90
+ - Kabyle
91
+ - Korean
92
+ - Dutch
93
+ - Norwegian
94
+ - Polish
95
+ - Portuguese
96
+ - Russian
97
+ - Swedish
98
+ - Thai
99
+ - Klingon
100
+ - Turkish
101
+ - Chinese
102
+
44
103
  ## Usage
45
104
 
46
105
  ### Detect Profanity
@@ -49,7 +108,8 @@ Please ensure that you have one of these Python versions installed before using
49
108
  from valx import detect_profanity
50
109
 
51
110
  # Detect profanity
52
- num_profanities = detect_profanity(sample_text, language='English')
111
+ results = detect_profanity(sample_text, language='English')
112
+ print("Profanity Evaluation Results", results)
53
113
  ```
54
114
 
55
115
  ### Remove Profanity
@@ -70,6 +130,9 @@ from valx import detect_sensitive_information
70
130
  detected_sensitive_info = detect_sensitive_information(sample_text)
71
131
  ```
72
132
 
133
+ > [!NOTE]
134
+ > We have updated this function, and it now includes an optional argument for `info_type`, which can be used to detect only specific types of sensitive information. It was also added to `remove_sensitive_information`.
135
+
73
136
  ### Remove Sensitive Information
74
137
 
75
138
  ```python
@@ -88,7 +151,14 @@ from valx import detect_hate_speech
88
151
  outcome_of_detection = detect_hate_speech("You are stupid.")
89
152
  ```
90
153
 
91
- > See our [official documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more examples on how to use ValX.*
154
+ > [!IMPORTANT]
155
+ > The model's possible outputs are:
156
+ > - `['Hate Speech']`: The text was flagged and contained hate speech.
157
+ > - `['Offensive Speech']`: The text was flagged and contained offensive speech.
158
+ > - `['No Hate and Offensive Speech']`: The text was not flagged for any hate speech or offensive speech.
159
+
160
+ > [!NOTE]
161
+ > See our [official documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more examples on how to use **ValX**.
92
162
 
93
163
  ## Contributing
94
164
 
@@ -5,7 +5,7 @@ with open("README.md", "r", encoding="utf-8") as fh:
5
5
 
6
6
  setup(
7
7
  name='valx',
8
- version='0.2.1',
8
+ version='0.2.3',
9
9
  author='Infinitode Pty Ltd',
10
10
  author_email='infinitode.ltd@gmail.com',
11
11
  description='An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.',
@@ -2754,11 +2754,14 @@ def detect_profanity(text_data, language='English'):
2754
2754
 
2755
2755
  Args:
2756
2756
  text_data (list): A list of strings representing the text data to analyze.
2757
- language (str): The language for which to detect profanity. Defaults to 'English'.
2758
- allowed_languages (list): A list of languages to allow for detection. If provided, only profanity words for these languages will be considered.
2759
-
2757
+ language (str): The language used to detect profanity. Defaults to 'English'. Available languages include: All, Arabic, Czech, Danish, German, English, Esperanto, Persian, Finnish, Filipino, French, French (CA), Hindi, Hungarian, Italian, Japanese, Kabyle, Korean, Dutch, Norwegian, Polish, Portuguese, Russian, Swedish, Thai, Klingon, Turkish, Chinese.
2760
2758
  Returns:
2761
- int: The total number of profanity detections.
2759
+ list: A list of dictionaries where each dictionary represents a detected instance of profanity.
2760
+ Each dictionary contains the following keys:
2761
+ - "Line" (int): The line number where the profanity was detected.
2762
+ - "Column" (int): The column number (position in the line) where the profanity starts.
2763
+ - "Word" (str): The detected profanity word.
2764
+ - "Language" (str): The language in which the profanity was detected.
2762
2765
  """
2763
2766
  profanity_keywords = load_profanity_words(language)
2764
2767
  detected = set() # Use a set to store unique occurrences
@@ -2774,12 +2777,19 @@ def detect_profanity(text_data, language='English'):
2774
2777
  for match in matches:
2775
2778
  detected.add((i + 1, match.start() + 1, language, profanity)) # Add to set
2776
2779
 
2780
+ detections = [] # Initialize detections as a list
2781
+
2777
2782
  if detected:
2778
- print(f"Profanity detected:")
2779
2783
  for line_num, col_num, lang, word in detected:
2780
- print(f"Language: {lang}, Line {line_num}, Column {col_num}: '{word}'")
2784
+ detection_info = {
2785
+ "Line": line_num,
2786
+ "Column": col_num,
2787
+ "Word": word,
2788
+ "Language": lang
2789
+ }
2790
+ detections.append(detection_info) # Append each detection to the list
2781
2791
 
2782
- return len(detected)
2792
+ return detections
2783
2793
 
2784
2794
 
2785
2795
  def remove_profanity(text_data, output_file=None, language='English'):
@@ -2815,85 +2825,68 @@ def remove_profanity(text_data, output_file=None, language='English'):
2815
2825
 
2816
2826
  return cleaned_data
2817
2827
 
2818
- # New functions for 1.5
2819
- def detect_sensitive_information(text_data):
2828
+ def detect_sensitive_information(text_data, info_type=["email", "phone", "credit_card", "ssn", "id", "address", "ip", "iban", "mrn", "icd10", "geo_coords", "username", "file_path", "bitcoin_wallet", "ethereum_wallet"]):
2820
2829
  """
2821
2830
  Detect sensitive information patterns in the provided text data.
2822
2831
 
2823
2832
  Args:
2824
2833
  text_data (list of str): A list of strings representing the text data to be analyzed.
2834
+ info_type (str or list of str, optional): One or more types of sensitive info to detect. Available types are: "email", "phone", "credit_card", "ssn", "id", "address", "ip", "iban", "mrn", "icd10", "geo_coords", "username", "file_path", "bitcoin_wallet", "ethereum_wallet". Uses all info types by default.
2825
2835
 
2826
2836
  Returns:
2827
2837
  list of tuple: A list of tuples containing detected sensitive information, each tuple
2828
2838
  representing (line number, column index, type, value).
2829
2839
  """
2840
+ if isinstance(info_type, str):
2841
+ info_type = [info_type] # Convert single string to list for consistent handling
2842
+
2843
+ # Dictionary to map info_type to regex patterns
2844
+ regex_patterns = {
2845
+ "email": r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b',
2846
+ "phone": r'\b(?:\d{3}[-.\s]??\d{3}[-.\s]??\d{4}|\(\d{3}\)\s*\d{3}[-.\s]??\d{4}|\d{3}[-.\s]??\d{4})\b',
2847
+ "credit_card": r'\b(?:\d[ -]*?){13,16}\b',
2848
+ "ssn": r'\b\d{3}[-]?\d{2}[-]?\d{4}\b',
2849
+ "id": r'\b[A-Za-z]{1,2}\d{6,9}\b',
2850
+ "address": r'\b\d+\s\w+\s\w+,\s\w+,\s\w+\b',
2851
+ "ip": r'\b(?:[0-9]{1,3}\.){3}[0-9]{1,3}\b',
2852
+ "iban": r'\b[A-Z]{2}[0-9]{2}[A-Z0-9]{1,30}\b',
2853
+ "mrn": r'\b\d{5,10}\b',
2854
+ "icd10": r'\b[A-TV-Z][0-9]{2}(?:\.[0-9A-TV-Z]{1,4})?\b',
2855
+ "geo_coords": r'\b[-+]?([1-8]?\d(\.\d+)?|90(\.0+)?),\s*[-+]?(180(\.0+)?|((1[0-7]\d)|([1-9]?\d))(\.\d+)?)\b',
2856
+ "username": r'@\w+',
2857
+ "file_path": r'([A-Za-z]:\\|/)?([-\w\s]+\\|/)*[-\w\s]+\.\w+',
2858
+ "bitcoin_wallet": r'\b[13][a-km-zA-HJ-NP-Z1-9]{25,34}\b',
2859
+ "ethereum_wallet": r'\b0x[a-fA-F0-9]{40}\b'
2860
+ }
2861
+
2830
2862
  sensitive_info = []
2831
2863
 
2832
- # Regular expression for detecting email addresses
2833
- email_regex = r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b'
2834
- # Regular expression for detecting phone numbers
2835
- phone_regex = r'\b(?:\d{3}[-.\s]??\d{3}[-.\s]??\d{4}|\(\d{3}\)\s*\d{3}[-.\s]??\d{4}|\d{3}[-.\s]??\d{4})\b'
2836
- # Regular expression for detecting credit card numbers (Visa, Mastercard, American Express, Discover)
2837
- credit_card_regex = r'\b(?:\d[ -]*?){13,16}\b'
2838
- # Regular expression for detecting social security numbers (SSN)
2839
- ssn_regex = r'\b\d{3}[-]?\d{2}[-]?\d{4}\b'
2840
- # Regular expression for detecting ID numbers (e.g., driver's license, passport)
2841
- id_number_regex = r'\b[A-Za-z]{1,2}\d{6,9}\b'
2842
- # Regular expression for detecting addresses (simple example)
2843
- address_regex = r'\b\d+\s\w+\s\w+,\s\w+,\s\w+\b'
2844
- # Regular expression for detecting IP addresses
2845
- ip_address_regex = r'\b(?:[0-9]{1,3}\.){3}[0-9]{1,3}\b'
2846
-
2864
+ # Loop through each line and check for sensitive info types based on specified `info_type`
2847
2865
  for i, line in enumerate(text_data):
2848
- # Detect email addresses
2849
- emails = re.findall(email_regex, line)
2850
- for email in emails:
2851
- sensitive_info.append((i + 1, line.find(email), 'Email', email))
2852
-
2853
- # Detect phone numbers
2854
- phones = re.findall(phone_regex, line)
2855
- for phone in phones:
2856
- sensitive_info.append((i + 1, line.find(phone), 'Phone', phone))
2857
-
2858
- # Detect credit card numbers
2859
- credit_cards = re.findall(credit_card_regex, line)
2860
- for card in credit_cards:
2861
- sensitive_info.append((i + 1, line.find(card), 'Credit Card', card))
2862
-
2863
- # Detect social security numbers
2864
- ssn_numbers = re.findall(ssn_regex, line)
2865
- for ssn in ssn_numbers:
2866
- sensitive_info.append((i + 1, line.find(ssn), 'SSN', ssn))
2867
-
2868
- # Detect ID numbers
2869
- id_numbers = re.findall(id_number_regex, line)
2870
- for id_number in id_numbers:
2871
- sensitive_info.append((i + 1, line.find(id_number), 'ID Number', id_number))
2872
-
2873
- # Detect addresses
2874
- addresses = re.findall(address_regex, line)
2875
- for address in addresses:
2876
- sensitive_info.append((i + 1, line.find(address), 'Address', address))
2877
-
2878
- # Detect IP addresses
2879
- ip_addresses = re.findall(ip_address_regex, line)
2880
- for ip_address in ip_addresses:
2881
- sensitive_info.append((i + 1, line.find(ip_address), 'IP Address', ip_address))
2882
-
2866
+ for type_ in info_type:
2867
+ pattern = regex_patterns.get(type_)
2868
+ if pattern:
2869
+ matches = re.finditer(pattern, line)
2870
+ for match in matches:
2871
+ sensitive_info.append((i + 1, match.start(), type_.capitalize(), match.group()))
2872
+
2883
2873
  return sensitive_info
2884
2874
 
2885
- def remove_sensitive_information(text_data, output_file=None):
2875
+ def remove_sensitive_information(text_data, output_file=None, info_type=["email", "phone", "credit_card", "ssn", "id", "address", "ip", "iban", "mrn", "icd10", "geo_coords", "username", "file_path", "bitcoin_wallet", "ethereum_wallet"]):
2886
2876
  """
2887
2877
  Remove sensitive information patterns from the provided text data.
2888
2878
 
2889
2879
  Args:
2890
2880
  text_data (list of str): A list of strings representing the text data to be cleaned.
2891
2881
  output_file (str, optional): Path to the output file where cleaned data will be saved.
2882
+ info_type (str or list of str, optional): One or more types of sensitive info to detect and remove. Available types are: "email", "phone", "credit_card", "ssn", "id", "address", "ip", "iban", "mrn", "icd10", "geo_coords", "username", "file_path", "bitcoin_wallet", "ethereum_wallet". Uses all info types by default.
2892
2883
 
2893
2884
  Returns:
2894
2885
  list of str: A list of strings representing the cleaned text data.
2895
2886
  """
2896
- sensitive_info = detect_sensitive_information(text_data)
2887
+ # Detect sensitive information
2888
+ sensitive_info = detect_sensitive_information(text_data, info_type=info_type)
2889
+
2897
2890
  cleaned_data = []
2898
2891
  for line in text_data:
2899
2892
  cleaned_line = line
@@ -2907,7 +2900,6 @@ def remove_sensitive_information(text_data, output_file=None):
2907
2900
  file.write('\n'.join(cleaned_data))
2908
2901
  return cleaned_data
2909
2902
 
2910
-
2911
2903
  # New version
2912
2904
  MODEL_DIR = os.path.join(os.path.dirname(__file__), 'models')
2913
2905
  DECISION_TREE_MODEL_PATH = os.path.join(MODEL_DIR, 'decision_tree_model.sav')
@@ -2927,9 +2919,6 @@ def detect_hate_speech(text):
2927
2919
  Returns:
2928
2920
  list of str: A list of strings representing the outcome of the detection.
2929
2921
  """
2930
- # if isinstance(text, str):
2931
- # text = [text]
2932
-
2933
2922
  text = cv.transform([text]).toarray()
2934
2923
  return model.predict((text))
2935
2924
 
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.1
2
2
  Name: valx
3
- Version: 0.2.1
3
+ Version: 0.2.3
4
4
  Summary: An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Now includes offensive language and hate speech detection using an AI model.
5
5
  Home-page: https://github.com/infinitode/valx
6
6
  Author: Infinitode Pty Ltd
@@ -27,7 +27,34 @@ Requires-Dist: scikit-learn==1.2.2
27
27
  ![License Compliance](https://img.shields.io/badge/license-compliance-brightgreen.svg)
28
28
  ![PyPI Version](https://img.shields.io/pypi/v/valx)
29
29
 
30
- An open-source Python library for data cleaning tasks. Includes profanity detection, and removal. Also now includes personal information detection and removal. Now includes hate speech and offensive language detection using AI.
30
+ An open-source Python library for data cleaning tasks. It includes functions for profanity detection, and removal, and detection and removal of personal information. Also includes hate speech and offensive language detection and removal, using AI.
31
+
32
+ > [!IMPORTANT]
33
+ > Please downgrade to `numpy` version `1.26.4`. Our ValX **DecisionTreeClassifier** AI model, relies on lower versions of `numpy`, because it was trained on these versions.
34
+ > For more information see: https://techoverflow.net/2024/07/23/how-to-fix-numpy-dtype-size-changed-may-indicate-binary-incompatibility-expected-96-from-c-header-got-88-from-pyobject/
35
+
36
+ ## Changes in 0.2.3
37
+
38
+ We have introduced a new optional `info_type` parameter into our `detect_sensitive_information`, and `remove_sensitive_information` functions, to allow you to have fine-grained control over what sensitive information you want to detect or remove.
39
+
40
+ Also introduced more detection patterns for other types of sensitive information, including:
41
+ - `"iban"`: International Bank Account Number.
42
+ - `"mrn"`: Medical Record Number (may not work correctly, depending on provider and country).
43
+ - `"icd10"`: International Classification of Diseases, Tenth Revision.
44
+ - `"geo_coords"`: Geo-coordinates (latitude and longitude in decimal degrees format).
45
+ - `"username"`: Username handles (@username).
46
+ - `"file_path"`: File paths (general patterns for both Windows and Unix paths).
47
+ - `"bitcoin_wallet"`: Cryptocurrency wallet address.
48
+ - `"ethereum_wallet"`: Cryptocurrency wallet addresses.
49
+
50
+ ## Changes in 0.2.2
51
+
52
+ We have refactored and changed the `detect_profanity` function:
53
+ - Removed unnecessary printing
54
+ - Now returns more information about each found profanity, including `Line`, `Column`, `Word`, and `Language`.
55
+
56
+ > [!NOTE]
57
+ > You can view [ValX's package documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more information on changes.
31
58
 
32
59
  ## Changes in 0.2.1
33
60
 
@@ -63,6 +90,38 @@ Please ensure that you have one of these Python versions installed before using
63
90
  - **Detect Hate Speech**: Detect hate speech or offensive speech in text, using AI.
64
91
  - **Remove Hate Speech**: Remove hate speech or offensive speech in text, using AI.
65
92
 
93
+ ### List of supported languages for profanity detection and removal
94
+ Below is a complete list of all the available supported languages for ValX's profanity detection and removal functions which are valid values for `language`:
95
+
96
+ - **All**
97
+ - Arabic
98
+ - Czech
99
+ - Danish
100
+ - German
101
+ - English
102
+ - Esperanto
103
+ - Persian
104
+ - Finnish
105
+ - Filipino
106
+ - French
107
+ - French (CA)
108
+ - Hindi
109
+ - Hungarian
110
+ - Italian
111
+ - Japanese
112
+ - Kabyle
113
+ - Korean
114
+ - Dutch
115
+ - Norwegian
116
+ - Polish
117
+ - Portuguese
118
+ - Russian
119
+ - Swedish
120
+ - Thai
121
+ - Klingon
122
+ - Turkish
123
+ - Chinese
124
+
66
125
  ## Usage
67
126
 
68
127
  ### Detect Profanity
@@ -71,7 +130,8 @@ Please ensure that you have one of these Python versions installed before using
71
130
  from valx import detect_profanity
72
131
 
73
132
  # Detect profanity
74
- num_profanities = detect_profanity(sample_text, language='English')
133
+ results = detect_profanity(sample_text, language='English')
134
+ print("Profanity Evaluation Results", results)
75
135
  ```
76
136
 
77
137
  ### Remove Profanity
@@ -92,6 +152,9 @@ from valx import detect_sensitive_information
92
152
  detected_sensitive_info = detect_sensitive_information(sample_text)
93
153
  ```
94
154
 
155
+ > [!NOTE]
156
+ > We have updated this function, and it now includes an optional argument for `info_type`, which can be used to detect only specific types of sensitive information. It was also added to `remove_sensitive_information`.
157
+
95
158
  ### Remove Sensitive Information
96
159
 
97
160
  ```python
@@ -110,7 +173,14 @@ from valx import detect_hate_speech
110
173
  outcome_of_detection = detect_hate_speech("You are stupid.")
111
174
  ```
112
175
 
113
- > See our [official documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more examples on how to use ValX.*
176
+ > [!IMPORTANT]
177
+ > The model's possible outputs are:
178
+ > - `['Hate Speech']`: The text was flagged and contained hate speech.
179
+ > - `['Offensive Speech']`: The text was flagged and contained offensive speech.
180
+ > - `['No Hate and Offensive Speech']`: The text was not flagged for any hate speech or offensive speech.
181
+
182
+ > [!NOTE]
183
+ > See our [official documentation](https://infinitode-docs.gitbook.io/documentation/package-documentation/valx-package-documentation) for more examples on how to use **ValX**.
114
184
 
115
185
  ## Contributing
116
186
 
File without changes
File without changes
File without changes
File without changes
File without changes