stream2sentence 0.2.4__tar.gz → 0.2.6__tar.gz

stream2sentence-0.2.6/PKG-INFO

@@ -0,0 +1,189 @@
+Metadata-Version: 2.1
+Name: stream2sentence
+Version: 0.2.6
+Summary: Real-time processing and delivery of sentences from a continuous stream of characters or text chunks.
+Home-page: https://github.com/KoljaB/stream2sentence
+Author: Kolja Beigel
+Author-email: kolja.beigel@web.de
+Keywords: realtime,text streaming,stream,sentence,sentence detection,sentence generation,tts,speech synthesis,nltk,text analysis,audio processing,boundary detection,sentence boundary detection
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.6
+Description-Content-Type: text/markdown
+Requires-Dist: nltk==3.9.1
+Requires-Dist: emoji==2.14.0
+Requires-Dist: stanza==1.9.2
+
+# Real-Time Sentence Detection
+
+Real-time processing and delivery of sentences from a continuous stream of characters or text chunks.
+
+> **Hint:** *If you're interested in state-of-the-art voice solutions you might also want to <strong>have a look at [Linguflex](https://github.com/KoljaB/Linguflex)</strong>, the original project from which stream2sentence is spun off. It lets you control your environment by speaking and is one of the most capable and sophisticated open-source assistants currently available.*
+
+## Table of Contents
+
+- [Features](#features)
+- [Installation](#installation)
+- [Usage](#usage)
+- [Configuration](#configuration)
+- [Contributing](#contributing)
+- [License](#license)
+
+## Features
+
+- Generates sentences from a stream of text in real-time.
+- Customizable to finetune/balance speed vs reliability.
+- Option to clean the output by removing links and emojis from the detected sentences.
+- Easy to configure and integrate.
+
+## Installation
+
+```bash
+pip install stream2sentence
+```
+
+## Usage
+
+Pass a generator of characters or text chunks to `generate_sentences()` to get a generator of sentences in return.
+
+Here's a basic example:
+
+```python
+from stream2sentence import generate_sentences
+
+# Dummy generator for demonstration
+def dummy_generator():
+    yield "This is a sentence. And here's another! Yet, "
+    yield "there's more. This ends now."
+
+for sentence in generate_sentences(dummy_generator()):
+    print(sentence)
+```
+
+This will output:
+```
+This is a sentence.
+And here's another!
+Yet, there's more.
+This ends now.
+```
+
+One main use case of this library is enable fast text to speech synthesis in the context of character feeds generated from large language models: this library enables fastest possible access to a complete sentence or sentence fragment (using the quick_yield_single_sentence_fragment flag) that then can be synthesized in realtime. The usage of this is demonstrated in the test_stream_from_llm.py file in the tests directory.
+
+## Configuration
+
+The `generate_sentences()` function offers various parameters to fine-tune its behavior:
+
+### Core Parameters
+
+- `generator: Iterator[str]`
+  - The primary input source, yielding chunks of text to be processed.
+  - Can be any iterator that emits text chunks of any size.
+
+- `context_size: int = 12`
+  - Number of characters considered for sentence boundary detection.
+  - Larger values improve accuracy but may increase latency.
+  - Default: 12 characters
+
+- `context_size_look_overhead: int = 12`
+  - Additional characters to examine beyond `context_size` for sentence splitting.
+  - Enhances sentence detection accuracy.
+  - Default: 12 characters
+
+- `minimum_sentence_length: int = 10`
+  - Minimum character count for a text chunk to be considered a sentence.
+  - Shorter fragments are buffered until this threshold is met.
+  - Default: 10 characters
+
+- `minimum_first_fragment_length: int = 10`
+  - Minimum character count required for the first sentence fragment.
+  - Ensures the initial output meets a specified length threshold.
+  - Default: 10 characters
+
+### Yield Control
+
+These parameters control how quickly and frequently the generator yields sentence fragments:
+
+- `quick_yield_single_sentence_fragment: bool = False`
+  - When True, yields the first fragment of the first sentence as quickly as possible.
+  - Useful for getting immediate output in real-time applications like speech synthesis.
+  - Default: False
+
+- `quick_yield_for_all_sentences: bool = False`
+  - When True, yields the first fragment of every sentence as quickly as possible.
+  - Extends the quick yield behavior to all sentences, not just the first one.
+  - Automatically sets `quick_yield_single_sentence_fragment` to True.
+  - Default: False
+
+- `quick_yield_every_fragment: bool = False`
+  - When True, yields every fragment of every sentence as quickly as possible.
+  - Provides the most granular output, yielding fragments as soon as they're detected.
+  - Automatically sets both `quick_yield_for_all_sentences` and `quick_yield_single_sentence_fragment` to True.
+  - Default: False
+
+### Text Cleanup
+
+- `cleanup_text_links: bool = False`
+  - When True, removes hyperlinks from the output sentences.
+  - Default: False
+
+- `cleanup_text_emojis: bool = False`
+  - When True, removes emoji characters from the output sentences.
+  - Default: False
+
+### Tokenization
+
+- `tokenize_sentences: Callable = None`
+  - Custom function for sentence tokenization.
+  - If None, uses the default tokenizer specified by `tokenizer`.
+  - Default: None
+
+- `tokenizer: str = "nltk"`
+  - Specifies the tokenizer to use. Options: "nltk" or "stanza"
+  - Default: "nltk"
+
+- `language: str = "en"`
+  - Language setting for the tokenizer.
+  - Use "en" for English or "multilingual" for Stanza tokenizer.
+  - Default: "en"
+
+### Debugging and Fine-tuning
+
+- `log_characters: bool = False`
+  - When True, logs each processed character to the console.
+  - Useful for debugging or monitoring real-time processing.
+  - Default: False
+
+- `sentence_fragment_delimiters: str = ".?!;:,\n…)]}。-"`
+  - Characters considered as potential sentence fragment delimiters.
+  - Used for quick yielding of sentence fragments.
+  - Default: ".?!;:,\n…)]}。-"
+
+- `full_sentence_delimiters: str = ".?!\n…。"`
+  - Characters considered as full sentence delimiters.
+  - Used for more definitive sentence boundary detection.
+  - Default: ".?!\n…。"
+
+- `force_first_fragment_after_words: int = 15`
+  - Forces the yield of the first sentence fragment after this many words.
+  - Ensures timely output even with long opening sentences.
+  - Default: 15 words
+
+## Contributing
+
+Any Contributions you make are welcome and **greatly appreciated**.
+
+1. **Fork** the Project.
+2. **Create** your Feature Branch (`git checkout -b feature/AmazingFeature`).
+3. **Commit** your Changes (`git commit -m 'Add some AmazingFeature'`).
+4. **Push** to the Branch (`git push origin feature/AmazingFeature`).
+5. **Open** a Pull Request.
+
+## License
+
+This project is licensed under the MIT License. For more details, see the [`LICENSE`](LICENSE) file.
+
+---
+
+Project created and maintained by [Kolja Beigel](https://github.com/KoljaB).

stream2sentence-0.2.6/README.md

@@ -0,0 +1,172 @@
+# Real-Time Sentence Detection
+
+Real-time processing and delivery of sentences from a continuous stream of characters or text chunks.
+
+> **Hint:** *If you're interested in state-of-the-art voice solutions you might also want to <strong>have a look at [Linguflex](https://github.com/KoljaB/Linguflex)</strong>, the original project from which stream2sentence is spun off. It lets you control your environment by speaking and is one of the most capable and sophisticated open-source assistants currently available.*
+
+## Table of Contents
+
+- [Features](#features)
+- [Installation](#installation)
+- [Usage](#usage)
+- [Configuration](#configuration)
+- [Contributing](#contributing)
+- [License](#license)
+
+## Features
+
+- Generates sentences from a stream of text in real-time.
+- Customizable to finetune/balance speed vs reliability.
+- Option to clean the output by removing links and emojis from the detected sentences.
+- Easy to configure and integrate.
+
+## Installation
+
+```bash
+pip install stream2sentence
+```
+
+## Usage
+
+Pass a generator of characters or text chunks to `generate_sentences()` to get a generator of sentences in return.
+
+Here's a basic example:
+
+```python
+from stream2sentence import generate_sentences
+
+# Dummy generator for demonstration
+def dummy_generator():
+    yield "This is a sentence. And here's another! Yet, "
+    yield "there's more. This ends now."
+
+for sentence in generate_sentences(dummy_generator()):
+    print(sentence)
+```
+
+This will output:
+```
+This is a sentence.
+And here's another!
+Yet, there's more.
+This ends now.
+```
+
+One main use case of this library is enable fast text to speech synthesis in the context of character feeds generated from large language models: this library enables fastest possible access to a complete sentence or sentence fragment (using the quick_yield_single_sentence_fragment flag) that then can be synthesized in realtime. The usage of this is demonstrated in the test_stream_from_llm.py file in the tests directory.
+
+## Configuration
+
+The `generate_sentences()` function offers various parameters to fine-tune its behavior:
+
+### Core Parameters
+
+- `generator: Iterator[str]`
+  - The primary input source, yielding chunks of text to be processed.
+  - Can be any iterator that emits text chunks of any size.
+
+- `context_size: int = 12`
+  - Number of characters considered for sentence boundary detection.
+  - Larger values improve accuracy but may increase latency.
+  - Default: 12 characters
+
+- `context_size_look_overhead: int = 12`
+  - Additional characters to examine beyond `context_size` for sentence splitting.
+  - Enhances sentence detection accuracy.
+  - Default: 12 characters
+
+- `minimum_sentence_length: int = 10`
+  - Minimum character count for a text chunk to be considered a sentence.
+  - Shorter fragments are buffered until this threshold is met.
+  - Default: 10 characters
+
+- `minimum_first_fragment_length: int = 10`
+  - Minimum character count required for the first sentence fragment.
+  - Ensures the initial output meets a specified length threshold.
+  - Default: 10 characters
+
+### Yield Control
+
+These parameters control how quickly and frequently the generator yields sentence fragments:
+
+- `quick_yield_single_sentence_fragment: bool = False`
+  - When True, yields the first fragment of the first sentence as quickly as possible.
+  - Useful for getting immediate output in real-time applications like speech synthesis.
+  - Default: False
+
+- `quick_yield_for_all_sentences: bool = False`
+  - When True, yields the first fragment of every sentence as quickly as possible.
+  - Extends the quick yield behavior to all sentences, not just the first one.
+  - Automatically sets `quick_yield_single_sentence_fragment` to True.
+  - Default: False
+
+- `quick_yield_every_fragment: bool = False`
+  - When True, yields every fragment of every sentence as quickly as possible.
+  - Provides the most granular output, yielding fragments as soon as they're detected.
+  - Automatically sets both `quick_yield_for_all_sentences` and `quick_yield_single_sentence_fragment` to True.
+  - Default: False
+
+### Text Cleanup
+
+- `cleanup_text_links: bool = False`
+  - When True, removes hyperlinks from the output sentences.
+  - Default: False
+
+- `cleanup_text_emojis: bool = False`
+  - When True, removes emoji characters from the output sentences.
+  - Default: False
+
+### Tokenization
+
+- `tokenize_sentences: Callable = None`
+  - Custom function for sentence tokenization.
+  - If None, uses the default tokenizer specified by `tokenizer`.
+  - Default: None
+
+- `tokenizer: str = "nltk"`
+  - Specifies the tokenizer to use. Options: "nltk" or "stanza"
+  - Default: "nltk"
+
+- `language: str = "en"`
+  - Language setting for the tokenizer.
+  - Use "en" for English or "multilingual" for Stanza tokenizer.
+  - Default: "en"
+
+### Debugging and Fine-tuning
+
+- `log_characters: bool = False`
+  - When True, logs each processed character to the console.
+  - Useful for debugging or monitoring real-time processing.
+  - Default: False
+
+- `sentence_fragment_delimiters: str = ".?!;:,\n…)]}。-"`
+  - Characters considered as potential sentence fragment delimiters.
+  - Used for quick yielding of sentence fragments.
+  - Default: ".?!;:,\n…)]}。-"
+
+- `full_sentence_delimiters: str = ".?!\n…。"`
+  - Characters considered as full sentence delimiters.
+  - Used for more definitive sentence boundary detection.
+  - Default: ".?!\n…。"
+
+- `force_first_fragment_after_words: int = 15`
+  - Forces the yield of the first sentence fragment after this many words.
+  - Ensures timely output even with long opening sentences.
+  - Default: 15 words
+
+## Contributing
+
+Any Contributions you make are welcome and **greatly appreciated**.
+
+1. **Fork** the Project.
+2. **Create** your Feature Branch (`git checkout -b feature/AmazingFeature`).
+3. **Commit** your Changes (`git commit -m 'Add some AmazingFeature'`).
+4. **Push** to the Branch (`git push origin feature/AmazingFeature`).
+5. **Open** a Pull Request.
+
+## License
+
+This project is licensed under the MIT License. For more details, see the [`LICENSE`](LICENSE) file.
+
+---
+
+Project created and maintained by [Kolja Beigel](https://github.com/KoljaB).

{stream2sentence-0.2.4 → stream2sentence-0.2.6}/setup.py

@@ -5,7 +5,7 @@ with open("README.md", "r") as fh:
 
 setuptools.setup(
     name="stream2sentence", 
-    version="0.2.4",
+    version="0.2.6",
     author="Kolja Beigel",
     author_email="kolja.beigel@web.de",
     description="Real-time processing and delivery of sentences from a continuous stream of characters or text chunks.",
@@ -20,9 +20,9 @@ setuptools.setup(
     ],
     python_requires='>=3.6',
     install_requires=[
-        'nltk==3.8.1',
-        'emoji==2.8.0',
-        'stanza==1.6.1'
+        'nltk==3.9.1',
+        'emoji==2.14.0',
+        'stanza==1.9.2'
     ],
     keywords='realtime, text streaming, stream, sentence, sentence detection, sentence generation, tts, speech synthesis, nltk, text analysis, audio processing, boundary detection, sentence boundary detection'
 )

{stream2sentence-0.2.4 → stream2sentence-0.2.6}/stream2sentence/stream2sentence.py

@@ -175,9 +175,12 @@ def init_tokenizer(tokenizer: str, language: str = "en"):
 
 def generate_sentences(generator: Iterator[str],
                        context_size: int = 12,
+                       context_size_look_overhead: int = 12,
                        minimum_sentence_length: int = 10,
-                       minimum_first_fragment_length=10,
+                       minimum_first_fragment_length: int = 10,
                        quick_yield_single_sentence_fragment: bool = False,
+                       quick_yield_for_all_sentences: bool = False,
+                       quick_yield_every_fragment : bool = False,
                        cleanup_text_links: bool = False,
                        cleanup_text_emojis: bool = False,
                        tokenize_sentences=None,
@@ -185,7 +188,9 @@ def generate_sentences(generator: Iterator[str],
                        language: str = "en",
                        log_characters: bool = False,
                        sentence_fragment_delimiters: str = ".?!;:,\n…)]}。-",
+                       full_sentence_delimiters: str = ".?!\n…。",                       
                        force_first_fragment_after_words=15,
+                       debug_mode=False
                        ) -> Iterator[str]:
     """
     Generates well-formed sentences from a stream of characters or text chunks
@@ -198,6 +203,9 @@ def generate_sentences(generator: Iterator[str],
           for sentence boundary detection. A larger context improves the
           accuracy of detecting sentence boundaries.
           Default is 12 characters.
+        context_size_look_overhead: The number of characters to look
+          over the context_size boundaries to detect sentence splitting
+          characters (improves sentence detection).
         minimum_sentence_length (int): The minimum number of characters a
           sentence must have. If a sentence is shorter, it will be
           concatenated with the following one, improving the overall
@@ -208,9 +216,15 @@ def generate_sentences(generator: Iterator[str],
           required for the first sentence fragment before yielding.
           Default is 10 characters.
         quick_yield_single_sentence_fragment (bool): If set to True, the
-          generator will yield the first sentence fragment as quickly as
+          generator will yield the first sentence first fragment as quickly as
           possible. This is particularly useful for real-time applications
           such as speech synthesis.
+        quick_yield_for_all_sentences (bool): If set to True, the
+          generator will yield every sentence first fragment as quickly as
+          possible (not only the first sentence first fragment)
+        quick_yield_every_fragment (bool): If set to True, the
+          generator not only yield every sentence first fragment, but also every
+          following fragment.
         cleanup_text_links (bool): If True, removes hyperlinks from the text
           stream to ensure clean output.
         cleanup_text_emojis (bool): If True, filters out emojis from the text
@@ -224,10 +238,13 @@ def generate_sentences(generator: Iterator[str],
         log_characters (bool): If True, logs each character to the console as
           they are processed.
         sentence_fragment_delimiters (str): A string of characters that are
-          considered sentence delimiters. Default is ".?!;:,\n…)]}。-".
+          considered sentence fragment delimiters. Default is ".?!;:,\n…)]}。-".
+        full_sentence_delimiters (str): A string of characters that are
+          considered full sentence delimiters. Default is ".?!\n…。".
         force_first_fragment_after_words (int): The number of words after
           which the first sentence fragment is forced to be yielded.
           Default is 15 words.
+        debug_mode (bool): If True, enables debug mode for logging.
 
     Yields:
         Iterator[str]: An iterator of complete sentences constructed from the
@@ -242,24 +259,38 @@ def generate_sentences(generator: Iterator[str],
       making it versatile for different types of text processing applications.
     """
 
+     # Initialize the tokenizer based on the specified tokenizer and language
     global current_tokenizer
     current_tokenizer = tokenizer
     init_tokenizer(current_tokenizer, language)
 
-    buffer = ''
-    is_first_sentence = True
+    buffer = ''  # Buffer to hold incoming text for processing
+    is_first_sentence = True   # Flag to identify the first sentence
     word_count = 0  # Initialize word count
+    last_delimiter_position = -1  # Position of last full sentence delimiter
 
+    # Adjust quick yield flags based on settings
+    if quick_yield_every_fragment:
+        quick_yield_for_all_sentences = True
+
+    if quick_yield_for_all_sentences:
+        quick_yield_single_sentence_fragment = True
+
+     # Process each character from the generator
     for char in _generate_characters(generator, log_characters):
 
         if char:
             buffer += char
             buffer = buffer.lstrip()
 
-            # Update word count
+            # Update word count on encountering space or sentence fragment delimiter
             if char.isspace() or char in sentence_fragment_delimiters:
                 word_count += 1
 
+            if debug_mode:
+                print("\033[36mDebug: Added char, buffer size: \"{}\"\033[0m".format(len(buffer)))
+
+            # Check conditions to yield first sentence fragment quickly
             if (is_first_sentence
                     and len(buffer) > minimum_first_fragment_length
                     and quick_yield_single_sentence_fragment):
@@ -271,30 +302,76 @@ def generate_sentences(generator: Iterator[str],
                         buffer,
                         cleanup_text_links,
                         cleanup_text_emojis)
+                    if debug_mode:
+                        print("\033[36mDebug: Yielding first sentence fragment: \"{}\"\033[0m".format(yield_text))                    
                     yield yield_text
                     buffer = ""
-                    is_first_sentence = False
+                    if not quick_yield_every_fragment:
+                        is_first_sentence = False
                     continue
 
-            # Check if minimum length reached
+             # Continue accumulating characters if buffer is under minimum sentence length
             if len(buffer) <= minimum_sentence_length + context_size:
                 continue
 
-            # Potential delimiter character has to be a bit away from the end
-            # of the buffer. For reliable sentence detection the engine needs
-            # enough context to work with
-            delimiter_char = buffer[-context_size]
-
-            if delimiter_char in sentence_fragment_delimiters:
-                sentences = _tokenize_sentences(buffer, tokenize_sentences)
+            # Update last delimiter position if a new delimiter is found
+            if char in full_sentence_delimiters:
+                last_delimiter_position = len(buffer) - 1
+
+            # Define context window for checking potential sentence boundaries
+            context_window_end_pos = len(buffer) - context_size - 1
+            context_window_start_pos = context_window_end_pos - context_size_look_overhead
+            if context_window_start_pos < 0:
+                context_window_start_pos = 0
+
+            # Tokenize sentences from buffer
+            sentences = _tokenize_sentences(buffer, tokenize_sentences)
+
+            if debug_mode:
+                print("\033[36mbuffer: \"{}\"\033[0m".format(buffer))
+                print("\033[36mlast_delimiter_position: {}\033[0m".format(last_delimiter_position))
+                print("\033[36mlen(sentences) > 2: {}\033[0m".format(len(sentences) > 2))
+                print("\033[36mcontext_window_start_pos: {}\033[0m".format(context_window_start_pos))
+                print("\033[36mcontext_window_end_pos: {}\033[0m".format(context_window_end_pos))
+
+            # Process and yield sentences based on conditions
+            if (len(sentences) > 2 or 
+                (last_delimiter_position >= 0 and 
+                context_window_start_pos <= last_delimiter_position <= context_window_end_pos)
+                ):
+                
                 if len(sentences) > 1:
-                    total_length_except_first = sum(len(sentence) for sentence in sentences[1:])
-                    if total_length_except_first >= context_size:
+                    total_length_except_last = sum(len(sentence) for sentence in sentences[:-1])
+                    if total_length_except_last >= minimum_sentence_length:
                         for sentence in sentences[:-1]:
-                            yield sentence
+                            yield_text = _clean_text(
+                                sentence,
+                                cleanup_text_links,
+                                cleanup_text_emojis)
+                            if debug_mode:
+                                print("\033[36mDebug: Yielding sentence: \"{}\"\033[0m".format(yield_text))
+                            yield yield_text
+                        if quick_yield_for_all_sentences:
+                            is_first_sentence = True
+
+                        # we need to remember if the buffer ends with space
+                        # - sentences returned by the tokenizers are rtrimmed
+                        # - this takes any blank spaces away from the last unfinshed sentence
+                        # - we have to work around this by re-adding the blank space in this case
+                        ends_with_space = buffer.endswith(" ")
+
+                        # set buffer to last unfinshed sentence returned by tokenizers
                         buffer = sentences[-1]
 
-    # Yield remaining buffer
+                        # reset the blank space if it was there:
+                        if ends_with_space:
+                            buffer += " "
+
+                        # reset the last delimiter position after yielding
+                        last_delimiter_position = -1 
+
+    
+    # Yield remaining buffer as final sentence(s)
     if buffer:
         sentences = _tokenize_sentences(buffer, tokenize_sentences)
         sentence_buffer = ""
@@ -304,6 +381,8 @@ def generate_sentences(generator: Iterator[str],
             if len(sentence_buffer) < minimum_sentence_length:
                 sentence_buffer += " "
                 continue
+            if debug_mode:
+                print("\033[36mDebug: Yielding final sentence(s): \"{}\"\033[0m".format(yield_text))
             yield_text = _clean_text(
                 sentence_buffer,
                 cleanup_text_links,
@@ -317,4 +396,6 @@ def generate_sentences(generator: Iterator[str],
                 sentence_buffer,
                 cleanup_text_links,
                 cleanup_text_emojis)
+            if debug_mode:
+                print("\033[36mDebug: Yielding remaining text: \"{}\"\033[0m".format(yield_text))
             yield yield_text