SqueakyCleanText 0.2.6__tar.gz → 0.3.0__tar.gz

SqueakyCleanText-0.3.0/PKG-INFO

@@ -0,0 +1,161 @@
+Metadata-Version: 2.1
+Name: SqueakyCleanText
+Version: 0.3.0
+Summary: A comprehensive text cleaning and preprocessing pipeline.
+Home-page: https://github.com/rhnfzl/SqueakyCleanText
+Author: Rehan Fazal
+License: MIT
+Keywords: text cleaning,text preprocessing,NLP,natural language processing
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Software Development :: Libraries
+Classifier: Topic :: Text Processing
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Provides-Extra: dev
+Provides-Extra: test
+License-File: LICENSE
+
+<div align="center">
+
+# SqueakyCleanText
+
+[![PyPI](https://img.shields.io/pypi/v/squeakycleantext.svg)](https://pypi.org/project/squeakycleantext/)
+[![PyPI - Downloads](https://img.shields.io/pypi/dm/squeakycleantext)](https://pypistats.org/packages/squeakycleantext)
+[![Python package](https://github.com/rhnfzl/SqueakyCleanText/actions/workflows/python-package.yml/badge.svg)](https://github.com/rhnfzl/SqueakyCleanText/actions/workflows/python-package.yml)
+[![Python Versions](https://img.shields.io/badge/Python-3.10%20|%203.11%20|%203.12-blue)](https://pypi.org/project/squeakycleantext/)
+[![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
+
+A comprehensive text cleaning and preprocessing pipeline for machine learning and NLP tasks.
+</div>
+
+In the world of machine learning and natural language processing, clean and well-structured text data is crucial for building effective downstream models and managing token limits in language models. 
+
+SqueakyCleanText simplifies the process by automatically addressing common text issues, ensuring your data is clean and well-structured with minimal effort on your part.
+
+### Key Features
+- **Encoding Issues**: Corrects text encoding problems and handles bad Unicode characters.
+- **HTML and URLs**: Removes or replaces HTML tags and URLs with configurable tokens.
+- **Contact Information**: Handles emails, phone numbers, and other contact details with customizable replacement tokens.
+- **Named Entity Recognition (NER)**:
+  - Multi-language support (English, Dutch, German, Spanish)
+  - Ensemble voting technique for improved accuracy
+  - Configurable confidence thresholds
+  - Efficient batch processing
+  - Automatic text chunking for long documents
+  - GPU acceleration support
+- **Text Normalization**:
+  - Removes isolated letters and symbols
+  - Normalizes whitespace
+  - Handles currency symbols
+  - Year detection and replacement
+  - Number standardization
+- **Language Support**:
+  - Automatic language detection
+  - Language-specific NER models
+  - Language-aware stopword removal
+- **Dual Output Formats**:
+  - Language Model format (preserves structure with tokens)
+  - Statistical Model format (optimized for classical ML)
+- **Performance Optimization**:
+  - Batch processing support
+  - Configurable batch sizes
+  - Memory-efficient processing of large texts
+  - GPU memory management
+
+![Default Flow of cleaning Text](resources/sct_flow.png)
+
+### Benefits
+
+#### For Language Models
+- Maintains text structure while anonymizing sensitive information
+- Configurable token replacements
+- Preserves context while removing noise
+- Handles long documents through intelligent chunking
+
+#### For Statistical Models
+- Removes stopwords and punctuation
+- Case normalization
+- Special symbol removal
+- Optimized for classification tasks
+
+#### Advanced NER Processing
+- Ensemble approach reduces missed entities
+- Language-specific models improve accuracy
+- Confidence thresholds for precision control
+- Efficient batch processing for large datasets
+- Automatic handling of long documents
+
+## Installation
+
+```sh
+pip install SqueakyCleanText
+```
+
+## Usage
+
+### Basic Usage
+```python
+from sct import sct
+
+# Initialize the TextCleaner
+sx = sct.TextCleaner()
+
+# Process single text
+text = "Hey John Doe, email me at john.doe@example.com"
+lm_text, stat_text, language = sx.process(text)
+
+# Process multiple texts efficiently
+texts = ["Text 1", "Text 2", "Text 3"]
+results = sx.process_batch(texts, batch_size=2)
+```
+
+### Advanced Configuration
+```python
+from sct import sct, config
+
+# Customize NER settings
+config.CHECK_NER_PROCESS = True
+config.NER_CONFIDENCE_THRESHOLD = 0.85
+config.POSITIONAL_TAGS = ['PER', 'LOC', 'ORG']
+
+# Customize replacement tokens
+config.REPLACE_WITH_URL = "<URL>"
+config.REPLACE_WITH_EMAIL = "<EMAIL>"
+config.REPLACE_WITH_PHONE_NUMBERS = "<PHONE>"
+
+# Set known language (skips detection)
+config.LANGUAGE = "ENGLISH"  # Options: ENGLISH, DUTCH, GERMAN, SPANISH
+
+# Initialize with custom settings
+sx = sct.TextCleaner()
+```
+
+## API
+
+### `sct.TextCleaner`
+
+#### `process(text: str) -> Tuple[str, str, str]`
+
+Processes the input text and returns a tuple containing:
+    - Cleaned text formatted for language models.
+    - Cleaned text formatted for statistical models (stopwords removed).
+    - Detected language of the text.
+
+## Contributing
+
+Contributions are welcome! Please feel free to submit a Pull Request or open an issue.
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+
+## Acknowledgements
+
+The package took inspirations from the following repo:
+
+- [clean-text](https://github.com/jfilter/clean-text)

SqueakyCleanText-0.3.0/README.md

@@ -0,0 +1,139 @@
+<div align="center">
+
+# SqueakyCleanText
+
+[![PyPI](https://img.shields.io/pypi/v/squeakycleantext.svg)](https://pypi.org/project/squeakycleantext/)
+[![PyPI - Downloads](https://img.shields.io/pypi/dm/squeakycleantext)](https://pypistats.org/packages/squeakycleantext)
+[![Python package](https://github.com/rhnfzl/SqueakyCleanText/actions/workflows/python-package.yml/badge.svg)](https://github.com/rhnfzl/SqueakyCleanText/actions/workflows/python-package.yml)
+[![Python Versions](https://img.shields.io/badge/Python-3.10%20|%203.11%20|%203.12-blue)](https://pypi.org/project/squeakycleantext/)
+[![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
+
+A comprehensive text cleaning and preprocessing pipeline for machine learning and NLP tasks.
+</div>
+
+In the world of machine learning and natural language processing, clean and well-structured text data is crucial for building effective downstream models and managing token limits in language models. 
+
+SqueakyCleanText simplifies the process by automatically addressing common text issues, ensuring your data is clean and well-structured with minimal effort on your part.
+
+### Key Features
+- **Encoding Issues**: Corrects text encoding problems and handles bad Unicode characters.
+- **HTML and URLs**: Removes or replaces HTML tags and URLs with configurable tokens.
+- **Contact Information**: Handles emails, phone numbers, and other contact details with customizable replacement tokens.
+- **Named Entity Recognition (NER)**:
+  - Multi-language support (English, Dutch, German, Spanish)
+  - Ensemble voting technique for improved accuracy
+  - Configurable confidence thresholds
+  - Efficient batch processing
+  - Automatic text chunking for long documents
+  - GPU acceleration support
+- **Text Normalization**:
+  - Removes isolated letters and symbols
+  - Normalizes whitespace
+  - Handles currency symbols
+  - Year detection and replacement
+  - Number standardization
+- **Language Support**:
+  - Automatic language detection
+  - Language-specific NER models
+  - Language-aware stopword removal
+- **Dual Output Formats**:
+  - Language Model format (preserves structure with tokens)
+  - Statistical Model format (optimized for classical ML)
+- **Performance Optimization**:
+  - Batch processing support
+  - Configurable batch sizes
+  - Memory-efficient processing of large texts
+  - GPU memory management
+
+![Default Flow of cleaning Text](resources/sct_flow.png)
+
+### Benefits
+
+#### For Language Models
+- Maintains text structure while anonymizing sensitive information
+- Configurable token replacements
+- Preserves context while removing noise
+- Handles long documents through intelligent chunking
+
+#### For Statistical Models
+- Removes stopwords and punctuation
+- Case normalization
+- Special symbol removal
+- Optimized for classification tasks
+
+#### Advanced NER Processing
+- Ensemble approach reduces missed entities
+- Language-specific models improve accuracy
+- Confidence thresholds for precision control
+- Efficient batch processing for large datasets
+- Automatic handling of long documents
+
+## Installation
+
+```sh
+pip install SqueakyCleanText
+```
+
+## Usage
+
+### Basic Usage
+```python
+from sct import sct
+
+# Initialize the TextCleaner
+sx = sct.TextCleaner()
+
+# Process single text
+text = "Hey John Doe, email me at john.doe@example.com"
+lm_text, stat_text, language = sx.process(text)
+
+# Process multiple texts efficiently
+texts = ["Text 1", "Text 2", "Text 3"]
+results = sx.process_batch(texts, batch_size=2)
+```
+
+### Advanced Configuration
+```python
+from sct import sct, config
+
+# Customize NER settings
+config.CHECK_NER_PROCESS = True
+config.NER_CONFIDENCE_THRESHOLD = 0.85
+config.POSITIONAL_TAGS = ['PER', 'LOC', 'ORG']
+
+# Customize replacement tokens
+config.REPLACE_WITH_URL = "<URL>"
+config.REPLACE_WITH_EMAIL = "<EMAIL>"
+config.REPLACE_WITH_PHONE_NUMBERS = "<PHONE>"
+
+# Set known language (skips detection)
+config.LANGUAGE = "ENGLISH"  # Options: ENGLISH, DUTCH, GERMAN, SPANISH
+
+# Initialize with custom settings
+sx = sct.TextCleaner()
+```
+
+## API
+
+### `sct.TextCleaner`
+
+#### `process(text: str) -> Tuple[str, str, str]`
+
+Processes the input text and returns a tuple containing:
+    - Cleaned text formatted for language models.
+    - Cleaned text formatted for statistical models (stopwords removed).
+    - Detected language of the text.
+
+## Contributing
+
+Contributions are welcome! Please feel free to submit a Pull Request or open an issue.
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+
+## Acknowledgements
+
+The package took inspirations from the following repo:
+
+- [clean-text](https://github.com/jfilter/clean-text)

SqueakyCleanText-0.3.0/SqueakyCleanText.egg-info/PKG-INFO

@@ -0,0 +1,161 @@
+Metadata-Version: 2.1
+Name: SqueakyCleanText
+Version: 0.3.0
+Summary: A comprehensive text cleaning and preprocessing pipeline.
+Home-page: https://github.com/rhnfzl/SqueakyCleanText
+Author: Rehan Fazal
+License: MIT
+Keywords: text cleaning,text preprocessing,NLP,natural language processing
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Software Development :: Libraries
+Classifier: Topic :: Text Processing
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Provides-Extra: dev
+Provides-Extra: test
+License-File: LICENSE
+
+<div align="center">
+
+# SqueakyCleanText
+
+[![PyPI](https://img.shields.io/pypi/v/squeakycleantext.svg)](https://pypi.org/project/squeakycleantext/)
+[![PyPI - Downloads](https://img.shields.io/pypi/dm/squeakycleantext)](https://pypistats.org/packages/squeakycleantext)
+[![Python package](https://github.com/rhnfzl/SqueakyCleanText/actions/workflows/python-package.yml/badge.svg)](https://github.com/rhnfzl/SqueakyCleanText/actions/workflows/python-package.yml)
+[![Python Versions](https://img.shields.io/badge/Python-3.10%20|%203.11%20|%203.12-blue)](https://pypi.org/project/squeakycleantext/)
+[![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
+
+A comprehensive text cleaning and preprocessing pipeline for machine learning and NLP tasks.
+</div>
+
+In the world of machine learning and natural language processing, clean and well-structured text data is crucial for building effective downstream models and managing token limits in language models. 
+
+SqueakyCleanText simplifies the process by automatically addressing common text issues, ensuring your data is clean and well-structured with minimal effort on your part.
+
+### Key Features
+- **Encoding Issues**: Corrects text encoding problems and handles bad Unicode characters.
+- **HTML and URLs**: Removes or replaces HTML tags and URLs with configurable tokens.
+- **Contact Information**: Handles emails, phone numbers, and other contact details with customizable replacement tokens.
+- **Named Entity Recognition (NER)**:
+  - Multi-language support (English, Dutch, German, Spanish)
+  - Ensemble voting technique for improved accuracy
+  - Configurable confidence thresholds
+  - Efficient batch processing
+  - Automatic text chunking for long documents
+  - GPU acceleration support
+- **Text Normalization**:
+  - Removes isolated letters and symbols
+  - Normalizes whitespace
+  - Handles currency symbols
+  - Year detection and replacement
+  - Number standardization
+- **Language Support**:
+  - Automatic language detection
+  - Language-specific NER models
+  - Language-aware stopword removal
+- **Dual Output Formats**:
+  - Language Model format (preserves structure with tokens)
+  - Statistical Model format (optimized for classical ML)
+- **Performance Optimization**:
+  - Batch processing support
+  - Configurable batch sizes
+  - Memory-efficient processing of large texts
+  - GPU memory management
+
+![Default Flow of cleaning Text](resources/sct_flow.png)
+
+### Benefits
+
+#### For Language Models
+- Maintains text structure while anonymizing sensitive information
+- Configurable token replacements
+- Preserves context while removing noise
+- Handles long documents through intelligent chunking
+
+#### For Statistical Models
+- Removes stopwords and punctuation
+- Case normalization
+- Special symbol removal
+- Optimized for classification tasks
+
+#### Advanced NER Processing
+- Ensemble approach reduces missed entities
+- Language-specific models improve accuracy
+- Confidence thresholds for precision control
+- Efficient batch processing for large datasets
+- Automatic handling of long documents
+
+## Installation
+
+```sh
+pip install SqueakyCleanText
+```
+
+## Usage
+
+### Basic Usage
+```python
+from sct import sct
+
+# Initialize the TextCleaner
+sx = sct.TextCleaner()
+
+# Process single text
+text = "Hey John Doe, email me at john.doe@example.com"
+lm_text, stat_text, language = sx.process(text)
+
+# Process multiple texts efficiently
+texts = ["Text 1", "Text 2", "Text 3"]
+results = sx.process_batch(texts, batch_size=2)
+```
+
+### Advanced Configuration
+```python
+from sct import sct, config
+
+# Customize NER settings
+config.CHECK_NER_PROCESS = True
+config.NER_CONFIDENCE_THRESHOLD = 0.85
+config.POSITIONAL_TAGS = ['PER', 'LOC', 'ORG']
+
+# Customize replacement tokens
+config.REPLACE_WITH_URL = "<URL>"
+config.REPLACE_WITH_EMAIL = "<EMAIL>"
+config.REPLACE_WITH_PHONE_NUMBERS = "<PHONE>"
+
+# Set known language (skips detection)
+config.LANGUAGE = "ENGLISH"  # Options: ENGLISH, DUTCH, GERMAN, SPANISH
+
+# Initialize with custom settings
+sx = sct.TextCleaner()
+```
+
+## API
+
+### `sct.TextCleaner`
+
+#### `process(text: str) -> Tuple[str, str, str]`
+
+Processes the input text and returns a tuple containing:
+    - Cleaned text formatted for language models.
+    - Cleaned text formatted for statistical models (stopwords removed).
+    - Detected language of the text.
+
+## Contributing
+
+Contributions are welcome! Please feel free to submit a Pull Request or open an issue.
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+
+## Acknowledgements
+
+The package took inspirations from the following repo:
+
+- [clean-text](https://github.com/jfilter/clean-text)

{SqueakyCleanText-0.2.6 → SqueakyCleanText-0.3.0}/SqueakyCleanText.egg-info/requires.txt

@@ -13,6 +13,7 @@ hypothesis==6.82.7
 faker==20.1.0
 flake8==6.1.0
 pytest==7.5.0
+timeout-decorator==0.5.0
 
 [test]
 coverage==7.3.1

{SqueakyCleanText-0.2.6 → SqueakyCleanText-0.3.0}/sct/sct.py

@@ -5,6 +5,7 @@ which is crucial for natural language processing tasks.
 """
 from sct import config
 from sct.utils import contact, datetime, ner, normtext, resources, special, stopwords
+from typing import List, Any
 
 class TextCleaner:
     
@@ -17,6 +18,7 @@ class TextCleaner:
         self.GeneralNER = ner.GeneralNER()
         self.pipeline = []
         self.language = None
+        self.batch_size = 8  # Default batch size for NER
         self.init_pipeline()
     
     def init_pipeline(self):
@@ -32,7 +34,6 @@ class TextCleaner:
             self.pipeline.append(self.fix_bad_unicode)
         if config.CHECK_TO_ASCII_UNICODE:
             self.pipeline.append(self.to_ascii_unicode)
-        
         if config.CHECK_REPLACE_HTML:
             self.pipeline.append(self.replace_html)
         if config.CHECK_REPLACE_URLS:
@@ -58,19 +59,56 @@ class TextCleaner:
         if config.CHECK_NORMALIZE_WHITESPACE:
             self.pipeline.append(self.normalize_whitespace)
     
-    def process(self, text):
-        text = str(text)
+    def process_batch(self, texts: List[str], batch_size: int = None) -> List[Any]:
+        """Process multiple texts efficiently in batches."""
+        if not texts:
+            return []
+            
+        results = []
+        batch_size = batch_size or self.batch_size
         
-        for step in self.pipeline:
-            text = step(text)
+        for text in texts:
+            # Validate input type and content
+            if not isinstance(text, str):
+                raise ValueError(f"Input must be string, got {type(text)}")
+            
+            # Handle empty text case
+            if not text or text.isspace():
+                results.append(("", "", None))
+                continue
+            
+            current_text = text  # No need for str() conversion now
+            
+            # Reset language for each text
+            self.language = None
+            
+            # Apply non-NER pipeline steps
+            for step in [s for s in self.pipeline if s != self.ner_process]:
+                current_text = step(current_text)
+            
+            # Batch NER processing if enabled
+            if config.CHECK_NER_PROCESS:
+                current_text = self.GeneralNER.ner_process(
+                    current_text,
+                    positional_tags=config.POSITIONAL_TAGS,
+                    ner_confidence_threshold=config.NER_CONFIDENCE_THRESHOLD,
+                    language=self.language
+                )
             
-        if config.CHECK_STATISTICAL_MODEL_PROCESSING:
-            stext = self.statistical_model_processing(text)
-            return text, stext, self.language
-        elif config.CHECK_DETECT_LANGUAGE:
-            return text, self.language
-        else:
-            return text
+            # Format results
+            if config.CHECK_STATISTICAL_MODEL_PROCESSING:
+                stext = self.statistical_model_processing(current_text)
+                results.append((current_text, stext, self.language))
+            elif config.CHECK_DETECT_LANGUAGE:
+                results.append((current_text, self.language))
+            else:
+                results.append(current_text)
+                
+        return results
+
+    def process(self, text: str) -> Any:
+        """Process a single text. Maintains backward compatibility."""
+        return self.process_batch([text])[0]
 
     def detect_language(self, text):
         self.language = str(resources.DETECTOR.detect_language_of(text)).split(".")[-1]