stream2sentence 0.2.4__tar.gz → 0.2.5__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,189 @@
1
+ Metadata-Version: 2.1
2
+ Name: stream2sentence
3
+ Version: 0.2.5
4
+ Summary: Real-time processing and delivery of sentences from a continuous stream of characters or text chunks.
5
+ Home-page: https://github.com/KoljaB/stream2sentence
6
+ Author: Kolja Beigel
7
+ Author-email: kolja.beigel@web.de
8
+ Keywords: realtime,text streaming,stream,sentence,sentence detection,sentence generation,tts,speech synthesis,nltk,text analysis,audio processing,boundary detection,sentence boundary detection
9
+ Classifier: Programming Language :: Python :: 3
10
+ Classifier: License :: OSI Approved :: MIT License
11
+ Classifier: Operating System :: OS Independent
12
+ Requires-Python: >=3.6
13
+ Description-Content-Type: text/markdown
14
+ Requires-Dist: nltk==3.8.1
15
+ Requires-Dist: emoji==2.8.0
16
+ Requires-Dist: stanza==1.6.1
17
+
18
+ # Real-Time Sentence Detection
19
+
20
+ Real-time processing and delivery of sentences from a continuous stream of characters or text chunks.
21
+
22
+ > **Hint:** *If you're interested in state-of-the-art voice solutions you might also want to <strong>have a look at [Linguflex](https://github.com/KoljaB/Linguflex)</strong>, the original project from which stream2sentence is spun off. It lets you control your environment by speaking and is one of the most capable and sophisticated open-source assistants currently available.*
23
+
24
+ ## Table of Contents
25
+
26
+ - [Features](#features)
27
+ - [Installation](#installation)
28
+ - [Usage](#usage)
29
+ - [Configuration](#configuration)
30
+ - [Contributing](#contributing)
31
+ - [License](#license)
32
+
33
+ ## Features
34
+
35
+ - Generates sentences from a stream of text in real-time.
36
+ - Customizable to finetune/balance speed vs reliability.
37
+ - Option to clean the output by removing links and emojis from the detected sentences.
38
+ - Easy to configure and integrate.
39
+
40
+ ## Installation
41
+
42
+ ```bash
43
+ pip install stream2sentence
44
+ ```
45
+
46
+ ## Usage
47
+
48
+ Pass a generator of characters or text chunks to `generate_sentences()` to get a generator of sentences in return.
49
+
50
+ Here's a basic example:
51
+
52
+ ```python
53
+ from stream2sentence import generate_sentences
54
+
55
+ # Dummy generator for demonstration
56
+ def dummy_generator():
57
+ yield "This is a sentence. And here's another! Yet, "
58
+ yield "there's more. This ends now."
59
+
60
+ for sentence in generate_sentences(dummy_generator()):
61
+ print(sentence)
62
+ ```
63
+
64
+ This will output:
65
+ ```
66
+ This is a sentence.
67
+ And here's another!
68
+ Yet, there's more.
69
+ This ends now.
70
+ ```
71
+
72
+ One main use case of this library is enable fast text to speech synthesis in the context of character feeds generated from large language models: this library enables fastest possible access to a complete sentence or sentence fragment (using the quick_yield_single_sentence_fragment flag) that then can be synthesized in realtime. The usage of this is demonstrated in the test_stream_from_llm.py file in the tests directory.
73
+
74
+ ## Configuration
75
+
76
+ The `generate_sentences()` function offers various parameters to fine-tune its behavior:
77
+
78
+ ### Core Parameters
79
+
80
+ - `generator: Iterator[str]`
81
+ - The primary input source, yielding chunks of text to be processed.
82
+ - Can be any iterator that emits text chunks of any size.
83
+
84
+ - `context_size: int = 12`
85
+ - Number of characters considered for sentence boundary detection.
86
+ - Larger values improve accuracy but may increase latency.
87
+ - Default: 12 characters
88
+
89
+ - `context_size_look_overhead: int = 12`
90
+ - Additional characters to examine beyond `context_size` for sentence splitting.
91
+ - Enhances sentence detection accuracy.
92
+ - Default: 12 characters
93
+
94
+ - `minimum_sentence_length: int = 10`
95
+ - Minimum character count for a text chunk to be considered a sentence.
96
+ - Shorter fragments are buffered until this threshold is met.
97
+ - Default: 10 characters
98
+
99
+ - `minimum_first_fragment_length: int = 10`
100
+ - Minimum character count required for the first sentence fragment.
101
+ - Ensures the initial output meets a specified length threshold.
102
+ - Default: 10 characters
103
+
104
+ ### Yield Control
105
+
106
+ These parameters control how quickly and frequently the generator yields sentence fragments:
107
+
108
+ - `quick_yield_single_sentence_fragment: bool = False`
109
+ - When True, yields the first fragment of the first sentence as quickly as possible.
110
+ - Useful for getting immediate output in real-time applications like speech synthesis.
111
+ - Default: False
112
+
113
+ - `quick_yield_for_all_sentences: bool = False`
114
+ - When True, yields the first fragment of every sentence as quickly as possible.
115
+ - Extends the quick yield behavior to all sentences, not just the first one.
116
+ - Automatically sets `quick_yield_single_sentence_fragment` to True.
117
+ - Default: False
118
+
119
+ - `quick_yield_every_fragment: bool = False`
120
+ - When True, yields every fragment of every sentence as quickly as possible.
121
+ - Provides the most granular output, yielding fragments as soon as they're detected.
122
+ - Automatically sets both `quick_yield_for_all_sentences` and `quick_yield_single_sentence_fragment` to True.
123
+ - Default: False
124
+
125
+ ### Text Cleanup
126
+
127
+ - `cleanup_text_links: bool = False`
128
+ - When True, removes hyperlinks from the output sentences.
129
+ - Default: False
130
+
131
+ - `cleanup_text_emojis: bool = False`
132
+ - When True, removes emoji characters from the output sentences.
133
+ - Default: False
134
+
135
+ ### Tokenization
136
+
137
+ - `tokenize_sentences: Callable = None`
138
+ - Custom function for sentence tokenization.
139
+ - If None, uses the default tokenizer specified by `tokenizer`.
140
+ - Default: None
141
+
142
+ - `tokenizer: str = "nltk"`
143
+ - Specifies the tokenizer to use. Options: "nltk" or "stanza"
144
+ - Default: "nltk"
145
+
146
+ - `language: str = "en"`
147
+ - Language setting for the tokenizer.
148
+ - Use "en" for English or "multilingual" for Stanza tokenizer.
149
+ - Default: "en"
150
+
151
+ ### Debugging and Fine-tuning
152
+
153
+ - `log_characters: bool = False`
154
+ - When True, logs each processed character to the console.
155
+ - Useful for debugging or monitoring real-time processing.
156
+ - Default: False
157
+
158
+ - `sentence_fragment_delimiters: str = ".?!;:,\n…)]}。-"`
159
+ - Characters considered as potential sentence fragment delimiters.
160
+ - Used for quick yielding of sentence fragments.
161
+ - Default: ".?!;:,\n…)]}。-"
162
+
163
+ - `full_sentence_delimiters: str = ".?!\n…。"`
164
+ - Characters considered as full sentence delimiters.
165
+ - Used for more definitive sentence boundary detection.
166
+ - Default: ".?!\n…。"
167
+
168
+ - `force_first_fragment_after_words: int = 15`
169
+ - Forces the yield of the first sentence fragment after this many words.
170
+ - Ensures timely output even with long opening sentences.
171
+ - Default: 15 words
172
+
173
+ ## Contributing
174
+
175
+ Any Contributions you make are welcome and **greatly appreciated**.
176
+
177
+ 1. **Fork** the Project.
178
+ 2. **Create** your Feature Branch (`git checkout -b feature/AmazingFeature`).
179
+ 3. **Commit** your Changes (`git commit -m 'Add some AmazingFeature'`).
180
+ 4. **Push** to the Branch (`git push origin feature/AmazingFeature`).
181
+ 5. **Open** a Pull Request.
182
+
183
+ ## License
184
+
185
+ This project is licensed under the MIT License. For more details, see the [`LICENSE`](LICENSE) file.
186
+
187
+ ---
188
+
189
+ Project created and maintained by [Kolja Beigel](https://github.com/KoljaB).
@@ -0,0 +1,172 @@
1
+ # Real-Time Sentence Detection
2
+
3
+ Real-time processing and delivery of sentences from a continuous stream of characters or text chunks.
4
+
5
+ > **Hint:** *If you're interested in state-of-the-art voice solutions you might also want to <strong>have a look at [Linguflex](https://github.com/KoljaB/Linguflex)</strong>, the original project from which stream2sentence is spun off. It lets you control your environment by speaking and is one of the most capable and sophisticated open-source assistants currently available.*
6
+
7
+ ## Table of Contents
8
+
9
+ - [Features](#features)
10
+ - [Installation](#installation)
11
+ - [Usage](#usage)
12
+ - [Configuration](#configuration)
13
+ - [Contributing](#contributing)
14
+ - [License](#license)
15
+
16
+ ## Features
17
+
18
+ - Generates sentences from a stream of text in real-time.
19
+ - Customizable to finetune/balance speed vs reliability.
20
+ - Option to clean the output by removing links and emojis from the detected sentences.
21
+ - Easy to configure and integrate.
22
+
23
+ ## Installation
24
+
25
+ ```bash
26
+ pip install stream2sentence
27
+ ```
28
+
29
+ ## Usage
30
+
31
+ Pass a generator of characters or text chunks to `generate_sentences()` to get a generator of sentences in return.
32
+
33
+ Here's a basic example:
34
+
35
+ ```python
36
+ from stream2sentence import generate_sentences
37
+
38
+ # Dummy generator for demonstration
39
+ def dummy_generator():
40
+ yield "This is a sentence. And here's another! Yet, "
41
+ yield "there's more. This ends now."
42
+
43
+ for sentence in generate_sentences(dummy_generator()):
44
+ print(sentence)
45
+ ```
46
+
47
+ This will output:
48
+ ```
49
+ This is a sentence.
50
+ And here's another!
51
+ Yet, there's more.
52
+ This ends now.
53
+ ```
54
+
55
+ One main use case of this library is enable fast text to speech synthesis in the context of character feeds generated from large language models: this library enables fastest possible access to a complete sentence or sentence fragment (using the quick_yield_single_sentence_fragment flag) that then can be synthesized in realtime. The usage of this is demonstrated in the test_stream_from_llm.py file in the tests directory.
56
+
57
+ ## Configuration
58
+
59
+ The `generate_sentences()` function offers various parameters to fine-tune its behavior:
60
+
61
+ ### Core Parameters
62
+
63
+ - `generator: Iterator[str]`
64
+ - The primary input source, yielding chunks of text to be processed.
65
+ - Can be any iterator that emits text chunks of any size.
66
+
67
+ - `context_size: int = 12`
68
+ - Number of characters considered for sentence boundary detection.
69
+ - Larger values improve accuracy but may increase latency.
70
+ - Default: 12 characters
71
+
72
+ - `context_size_look_overhead: int = 12`
73
+ - Additional characters to examine beyond `context_size` for sentence splitting.
74
+ - Enhances sentence detection accuracy.
75
+ - Default: 12 characters
76
+
77
+ - `minimum_sentence_length: int = 10`
78
+ - Minimum character count for a text chunk to be considered a sentence.
79
+ - Shorter fragments are buffered until this threshold is met.
80
+ - Default: 10 characters
81
+
82
+ - `minimum_first_fragment_length: int = 10`
83
+ - Minimum character count required for the first sentence fragment.
84
+ - Ensures the initial output meets a specified length threshold.
85
+ - Default: 10 characters
86
+
87
+ ### Yield Control
88
+
89
+ These parameters control how quickly and frequently the generator yields sentence fragments:
90
+
91
+ - `quick_yield_single_sentence_fragment: bool = False`
92
+ - When True, yields the first fragment of the first sentence as quickly as possible.
93
+ - Useful for getting immediate output in real-time applications like speech synthesis.
94
+ - Default: False
95
+
96
+ - `quick_yield_for_all_sentences: bool = False`
97
+ - When True, yields the first fragment of every sentence as quickly as possible.
98
+ - Extends the quick yield behavior to all sentences, not just the first one.
99
+ - Automatically sets `quick_yield_single_sentence_fragment` to True.
100
+ - Default: False
101
+
102
+ - `quick_yield_every_fragment: bool = False`
103
+ - When True, yields every fragment of every sentence as quickly as possible.
104
+ - Provides the most granular output, yielding fragments as soon as they're detected.
105
+ - Automatically sets both `quick_yield_for_all_sentences` and `quick_yield_single_sentence_fragment` to True.
106
+ - Default: False
107
+
108
+ ### Text Cleanup
109
+
110
+ - `cleanup_text_links: bool = False`
111
+ - When True, removes hyperlinks from the output sentences.
112
+ - Default: False
113
+
114
+ - `cleanup_text_emojis: bool = False`
115
+ - When True, removes emoji characters from the output sentences.
116
+ - Default: False
117
+
118
+ ### Tokenization
119
+
120
+ - `tokenize_sentences: Callable = None`
121
+ - Custom function for sentence tokenization.
122
+ - If None, uses the default tokenizer specified by `tokenizer`.
123
+ - Default: None
124
+
125
+ - `tokenizer: str = "nltk"`
126
+ - Specifies the tokenizer to use. Options: "nltk" or "stanza"
127
+ - Default: "nltk"
128
+
129
+ - `language: str = "en"`
130
+ - Language setting for the tokenizer.
131
+ - Use "en" for English or "multilingual" for Stanza tokenizer.
132
+ - Default: "en"
133
+
134
+ ### Debugging and Fine-tuning
135
+
136
+ - `log_characters: bool = False`
137
+ - When True, logs each processed character to the console.
138
+ - Useful for debugging or monitoring real-time processing.
139
+ - Default: False
140
+
141
+ - `sentence_fragment_delimiters: str = ".?!;:,\n…)]}。-"`
142
+ - Characters considered as potential sentence fragment delimiters.
143
+ - Used for quick yielding of sentence fragments.
144
+ - Default: ".?!;:,\n…)]}。-"
145
+
146
+ - `full_sentence_delimiters: str = ".?!\n…。"`
147
+ - Characters considered as full sentence delimiters.
148
+ - Used for more definitive sentence boundary detection.
149
+ - Default: ".?!\n…。"
150
+
151
+ - `force_first_fragment_after_words: int = 15`
152
+ - Forces the yield of the first sentence fragment after this many words.
153
+ - Ensures timely output even with long opening sentences.
154
+ - Default: 15 words
155
+
156
+ ## Contributing
157
+
158
+ Any Contributions you make are welcome and **greatly appreciated**.
159
+
160
+ 1. **Fork** the Project.
161
+ 2. **Create** your Feature Branch (`git checkout -b feature/AmazingFeature`).
162
+ 3. **Commit** your Changes (`git commit -m 'Add some AmazingFeature'`).
163
+ 4. **Push** to the Branch (`git push origin feature/AmazingFeature`).
164
+ 5. **Open** a Pull Request.
165
+
166
+ ## License
167
+
168
+ This project is licensed under the MIT License. For more details, see the [`LICENSE`](LICENSE) file.
169
+
170
+ ---
171
+
172
+ Project created and maintained by [Kolja Beigel](https://github.com/KoljaB).
@@ -5,7 +5,7 @@ with open("README.md", "r") as fh:
5
5
 
6
6
  setuptools.setup(
7
7
  name="stream2sentence",
8
- version="0.2.4",
8
+ version="0.2.5",
9
9
  author="Kolja Beigel",
10
10
  author_email="kolja.beigel@web.de",
11
11
  description="Real-time processing and delivery of sentences from a continuous stream of characters or text chunks.",