stream2sentence 0.2.4__tar.gz → 0.2.5__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- stream2sentence-0.2.5/PKG-INFO +189 -0
- stream2sentence-0.2.5/README.md +172 -0
- {stream2sentence-0.2.4 → stream2sentence-0.2.5}/setup.py +1 -1
- stream2sentence-0.2.5/stream2sentence/stream2sentence - Kopie (2).py +403 -0
- {stream2sentence-0.2.4 → stream2sentence-0.2.5}/stream2sentence/stream2sentence.py +52 -12
- stream2sentence-0.2.5/stream2sentence.egg-info/PKG-INFO +189 -0
- {stream2sentence-0.2.4 → stream2sentence-0.2.5}/stream2sentence.egg-info/SOURCES.txt +1 -0
- stream2sentence-0.2.4/PKG-INFO +0 -138
- stream2sentence-0.2.4/README.md +0 -121
- stream2sentence-0.2.4/stream2sentence.egg-info/PKG-INFO +0 -138
- {stream2sentence-0.2.4 → stream2sentence-0.2.5}/setup.cfg +0 -0
- {stream2sentence-0.2.4 → stream2sentence-0.2.5}/stream2sentence/__init__.py +0 -0
- {stream2sentence-0.2.4 → stream2sentence-0.2.5}/stream2sentence/stream2sentence - Kopie.py +0 -0
- {stream2sentence-0.2.4 → stream2sentence-0.2.5}/stream2sentence.egg-info/dependency_links.txt +0 -0
- {stream2sentence-0.2.4 → stream2sentence-0.2.5}/stream2sentence.egg-info/requires.txt +0 -0
- {stream2sentence-0.2.4 → stream2sentence-0.2.5}/stream2sentence.egg-info/top_level.txt +0 -0
- {stream2sentence-0.2.4 → stream2sentence-0.2.5}/tests/test_stream2sentence.py +0 -0
- {stream2sentence-0.2.4 → stream2sentence-0.2.5}/tests/test_stream_from_llm.py +0 -0
- {stream2sentence-0.2.4 → stream2sentence-0.2.5}/tests/test_stream_from_llm_old_api.py +0 -0
|
@@ -0,0 +1,189 @@
|
|
|
1
|
+
Metadata-Version: 2.1
|
|
2
|
+
Name: stream2sentence
|
|
3
|
+
Version: 0.2.5
|
|
4
|
+
Summary: Real-time processing and delivery of sentences from a continuous stream of characters or text chunks.
|
|
5
|
+
Home-page: https://github.com/KoljaB/stream2sentence
|
|
6
|
+
Author: Kolja Beigel
|
|
7
|
+
Author-email: kolja.beigel@web.de
|
|
8
|
+
Keywords: realtime,text streaming,stream,sentence,sentence detection,sentence generation,tts,speech synthesis,nltk,text analysis,audio processing,boundary detection,sentence boundary detection
|
|
9
|
+
Classifier: Programming Language :: Python :: 3
|
|
10
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
11
|
+
Classifier: Operating System :: OS Independent
|
|
12
|
+
Requires-Python: >=3.6
|
|
13
|
+
Description-Content-Type: text/markdown
|
|
14
|
+
Requires-Dist: nltk==3.8.1
|
|
15
|
+
Requires-Dist: emoji==2.8.0
|
|
16
|
+
Requires-Dist: stanza==1.6.1
|
|
17
|
+
|
|
18
|
+
# Real-Time Sentence Detection
|
|
19
|
+
|
|
20
|
+
Real-time processing and delivery of sentences from a continuous stream of characters or text chunks.
|
|
21
|
+
|
|
22
|
+
> **Hint:** *If you're interested in state-of-the-art voice solutions you might also want to <strong>have a look at [Linguflex](https://github.com/KoljaB/Linguflex)</strong>, the original project from which stream2sentence is spun off. It lets you control your environment by speaking and is one of the most capable and sophisticated open-source assistants currently available.*
|
|
23
|
+
|
|
24
|
+
## Table of Contents
|
|
25
|
+
|
|
26
|
+
- [Features](#features)
|
|
27
|
+
- [Installation](#installation)
|
|
28
|
+
- [Usage](#usage)
|
|
29
|
+
- [Configuration](#configuration)
|
|
30
|
+
- [Contributing](#contributing)
|
|
31
|
+
- [License](#license)
|
|
32
|
+
|
|
33
|
+
## Features
|
|
34
|
+
|
|
35
|
+
- Generates sentences from a stream of text in real-time.
|
|
36
|
+
- Customizable to finetune/balance speed vs reliability.
|
|
37
|
+
- Option to clean the output by removing links and emojis from the detected sentences.
|
|
38
|
+
- Easy to configure and integrate.
|
|
39
|
+
|
|
40
|
+
## Installation
|
|
41
|
+
|
|
42
|
+
```bash
|
|
43
|
+
pip install stream2sentence
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
## Usage
|
|
47
|
+
|
|
48
|
+
Pass a generator of characters or text chunks to `generate_sentences()` to get a generator of sentences in return.
|
|
49
|
+
|
|
50
|
+
Here's a basic example:
|
|
51
|
+
|
|
52
|
+
```python
|
|
53
|
+
from stream2sentence import generate_sentences
|
|
54
|
+
|
|
55
|
+
# Dummy generator for demonstration
|
|
56
|
+
def dummy_generator():
|
|
57
|
+
yield "This is a sentence. And here's another! Yet, "
|
|
58
|
+
yield "there's more. This ends now."
|
|
59
|
+
|
|
60
|
+
for sentence in generate_sentences(dummy_generator()):
|
|
61
|
+
print(sentence)
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
This will output:
|
|
65
|
+
```
|
|
66
|
+
This is a sentence.
|
|
67
|
+
And here's another!
|
|
68
|
+
Yet, there's more.
|
|
69
|
+
This ends now.
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
One main use case of this library is enable fast text to speech synthesis in the context of character feeds generated from large language models: this library enables fastest possible access to a complete sentence or sentence fragment (using the quick_yield_single_sentence_fragment flag) that then can be synthesized in realtime. The usage of this is demonstrated in the test_stream_from_llm.py file in the tests directory.
|
|
73
|
+
|
|
74
|
+
## Configuration
|
|
75
|
+
|
|
76
|
+
The `generate_sentences()` function offers various parameters to fine-tune its behavior:
|
|
77
|
+
|
|
78
|
+
### Core Parameters
|
|
79
|
+
|
|
80
|
+
- `generator: Iterator[str]`
|
|
81
|
+
- The primary input source, yielding chunks of text to be processed.
|
|
82
|
+
- Can be any iterator that emits text chunks of any size.
|
|
83
|
+
|
|
84
|
+
- `context_size: int = 12`
|
|
85
|
+
- Number of characters considered for sentence boundary detection.
|
|
86
|
+
- Larger values improve accuracy but may increase latency.
|
|
87
|
+
- Default: 12 characters
|
|
88
|
+
|
|
89
|
+
- `context_size_look_overhead: int = 12`
|
|
90
|
+
- Additional characters to examine beyond `context_size` for sentence splitting.
|
|
91
|
+
- Enhances sentence detection accuracy.
|
|
92
|
+
- Default: 12 characters
|
|
93
|
+
|
|
94
|
+
- `minimum_sentence_length: int = 10`
|
|
95
|
+
- Minimum character count for a text chunk to be considered a sentence.
|
|
96
|
+
- Shorter fragments are buffered until this threshold is met.
|
|
97
|
+
- Default: 10 characters
|
|
98
|
+
|
|
99
|
+
- `minimum_first_fragment_length: int = 10`
|
|
100
|
+
- Minimum character count required for the first sentence fragment.
|
|
101
|
+
- Ensures the initial output meets a specified length threshold.
|
|
102
|
+
- Default: 10 characters
|
|
103
|
+
|
|
104
|
+
### Yield Control
|
|
105
|
+
|
|
106
|
+
These parameters control how quickly and frequently the generator yields sentence fragments:
|
|
107
|
+
|
|
108
|
+
- `quick_yield_single_sentence_fragment: bool = False`
|
|
109
|
+
- When True, yields the first fragment of the first sentence as quickly as possible.
|
|
110
|
+
- Useful for getting immediate output in real-time applications like speech synthesis.
|
|
111
|
+
- Default: False
|
|
112
|
+
|
|
113
|
+
- `quick_yield_for_all_sentences: bool = False`
|
|
114
|
+
- When True, yields the first fragment of every sentence as quickly as possible.
|
|
115
|
+
- Extends the quick yield behavior to all sentences, not just the first one.
|
|
116
|
+
- Automatically sets `quick_yield_single_sentence_fragment` to True.
|
|
117
|
+
- Default: False
|
|
118
|
+
|
|
119
|
+
- `quick_yield_every_fragment: bool = False`
|
|
120
|
+
- When True, yields every fragment of every sentence as quickly as possible.
|
|
121
|
+
- Provides the most granular output, yielding fragments as soon as they're detected.
|
|
122
|
+
- Automatically sets both `quick_yield_for_all_sentences` and `quick_yield_single_sentence_fragment` to True.
|
|
123
|
+
- Default: False
|
|
124
|
+
|
|
125
|
+
### Text Cleanup
|
|
126
|
+
|
|
127
|
+
- `cleanup_text_links: bool = False`
|
|
128
|
+
- When True, removes hyperlinks from the output sentences.
|
|
129
|
+
- Default: False
|
|
130
|
+
|
|
131
|
+
- `cleanup_text_emojis: bool = False`
|
|
132
|
+
- When True, removes emoji characters from the output sentences.
|
|
133
|
+
- Default: False
|
|
134
|
+
|
|
135
|
+
### Tokenization
|
|
136
|
+
|
|
137
|
+
- `tokenize_sentences: Callable = None`
|
|
138
|
+
- Custom function for sentence tokenization.
|
|
139
|
+
- If None, uses the default tokenizer specified by `tokenizer`.
|
|
140
|
+
- Default: None
|
|
141
|
+
|
|
142
|
+
- `tokenizer: str = "nltk"`
|
|
143
|
+
- Specifies the tokenizer to use. Options: "nltk" or "stanza"
|
|
144
|
+
- Default: "nltk"
|
|
145
|
+
|
|
146
|
+
- `language: str = "en"`
|
|
147
|
+
- Language setting for the tokenizer.
|
|
148
|
+
- Use "en" for English or "multilingual" for Stanza tokenizer.
|
|
149
|
+
- Default: "en"
|
|
150
|
+
|
|
151
|
+
### Debugging and Fine-tuning
|
|
152
|
+
|
|
153
|
+
- `log_characters: bool = False`
|
|
154
|
+
- When True, logs each processed character to the console.
|
|
155
|
+
- Useful for debugging or monitoring real-time processing.
|
|
156
|
+
- Default: False
|
|
157
|
+
|
|
158
|
+
- `sentence_fragment_delimiters: str = ".?!;:,\n…)]}。-"`
|
|
159
|
+
- Characters considered as potential sentence fragment delimiters.
|
|
160
|
+
- Used for quick yielding of sentence fragments.
|
|
161
|
+
- Default: ".?!;:,\n…)]}。-"
|
|
162
|
+
|
|
163
|
+
- `full_sentence_delimiters: str = ".?!\n…。"`
|
|
164
|
+
- Characters considered as full sentence delimiters.
|
|
165
|
+
- Used for more definitive sentence boundary detection.
|
|
166
|
+
- Default: ".?!\n…。"
|
|
167
|
+
|
|
168
|
+
- `force_first_fragment_after_words: int = 15`
|
|
169
|
+
- Forces the yield of the first sentence fragment after this many words.
|
|
170
|
+
- Ensures timely output even with long opening sentences.
|
|
171
|
+
- Default: 15 words
|
|
172
|
+
|
|
173
|
+
## Contributing
|
|
174
|
+
|
|
175
|
+
Any Contributions you make are welcome and **greatly appreciated**.
|
|
176
|
+
|
|
177
|
+
1. **Fork** the Project.
|
|
178
|
+
2. **Create** your Feature Branch (`git checkout -b feature/AmazingFeature`).
|
|
179
|
+
3. **Commit** your Changes (`git commit -m 'Add some AmazingFeature'`).
|
|
180
|
+
4. **Push** to the Branch (`git push origin feature/AmazingFeature`).
|
|
181
|
+
5. **Open** a Pull Request.
|
|
182
|
+
|
|
183
|
+
## License
|
|
184
|
+
|
|
185
|
+
This project is licensed under the MIT License. For more details, see the [`LICENSE`](LICENSE) file.
|
|
186
|
+
|
|
187
|
+
---
|
|
188
|
+
|
|
189
|
+
Project created and maintained by [Kolja Beigel](https://github.com/KoljaB).
|
|
@@ -0,0 +1,172 @@
|
|
|
1
|
+
# Real-Time Sentence Detection
|
|
2
|
+
|
|
3
|
+
Real-time processing and delivery of sentences from a continuous stream of characters or text chunks.
|
|
4
|
+
|
|
5
|
+
> **Hint:** *If you're interested in state-of-the-art voice solutions you might also want to <strong>have a look at [Linguflex](https://github.com/KoljaB/Linguflex)</strong>, the original project from which stream2sentence is spun off. It lets you control your environment by speaking and is one of the most capable and sophisticated open-source assistants currently available.*
|
|
6
|
+
|
|
7
|
+
## Table of Contents
|
|
8
|
+
|
|
9
|
+
- [Features](#features)
|
|
10
|
+
- [Installation](#installation)
|
|
11
|
+
- [Usage](#usage)
|
|
12
|
+
- [Configuration](#configuration)
|
|
13
|
+
- [Contributing](#contributing)
|
|
14
|
+
- [License](#license)
|
|
15
|
+
|
|
16
|
+
## Features
|
|
17
|
+
|
|
18
|
+
- Generates sentences from a stream of text in real-time.
|
|
19
|
+
- Customizable to finetune/balance speed vs reliability.
|
|
20
|
+
- Option to clean the output by removing links and emojis from the detected sentences.
|
|
21
|
+
- Easy to configure and integrate.
|
|
22
|
+
|
|
23
|
+
## Installation
|
|
24
|
+
|
|
25
|
+
```bash
|
|
26
|
+
pip install stream2sentence
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
## Usage
|
|
30
|
+
|
|
31
|
+
Pass a generator of characters or text chunks to `generate_sentences()` to get a generator of sentences in return.
|
|
32
|
+
|
|
33
|
+
Here's a basic example:
|
|
34
|
+
|
|
35
|
+
```python
|
|
36
|
+
from stream2sentence import generate_sentences
|
|
37
|
+
|
|
38
|
+
# Dummy generator for demonstration
|
|
39
|
+
def dummy_generator():
|
|
40
|
+
yield "This is a sentence. And here's another! Yet, "
|
|
41
|
+
yield "there's more. This ends now."
|
|
42
|
+
|
|
43
|
+
for sentence in generate_sentences(dummy_generator()):
|
|
44
|
+
print(sentence)
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
This will output:
|
|
48
|
+
```
|
|
49
|
+
This is a sentence.
|
|
50
|
+
And here's another!
|
|
51
|
+
Yet, there's more.
|
|
52
|
+
This ends now.
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
One main use case of this library is enable fast text to speech synthesis in the context of character feeds generated from large language models: this library enables fastest possible access to a complete sentence or sentence fragment (using the quick_yield_single_sentence_fragment flag) that then can be synthesized in realtime. The usage of this is demonstrated in the test_stream_from_llm.py file in the tests directory.
|
|
56
|
+
|
|
57
|
+
## Configuration
|
|
58
|
+
|
|
59
|
+
The `generate_sentences()` function offers various parameters to fine-tune its behavior:
|
|
60
|
+
|
|
61
|
+
### Core Parameters
|
|
62
|
+
|
|
63
|
+
- `generator: Iterator[str]`
|
|
64
|
+
- The primary input source, yielding chunks of text to be processed.
|
|
65
|
+
- Can be any iterator that emits text chunks of any size.
|
|
66
|
+
|
|
67
|
+
- `context_size: int = 12`
|
|
68
|
+
- Number of characters considered for sentence boundary detection.
|
|
69
|
+
- Larger values improve accuracy but may increase latency.
|
|
70
|
+
- Default: 12 characters
|
|
71
|
+
|
|
72
|
+
- `context_size_look_overhead: int = 12`
|
|
73
|
+
- Additional characters to examine beyond `context_size` for sentence splitting.
|
|
74
|
+
- Enhances sentence detection accuracy.
|
|
75
|
+
- Default: 12 characters
|
|
76
|
+
|
|
77
|
+
- `minimum_sentence_length: int = 10`
|
|
78
|
+
- Minimum character count for a text chunk to be considered a sentence.
|
|
79
|
+
- Shorter fragments are buffered until this threshold is met.
|
|
80
|
+
- Default: 10 characters
|
|
81
|
+
|
|
82
|
+
- `minimum_first_fragment_length: int = 10`
|
|
83
|
+
- Minimum character count required for the first sentence fragment.
|
|
84
|
+
- Ensures the initial output meets a specified length threshold.
|
|
85
|
+
- Default: 10 characters
|
|
86
|
+
|
|
87
|
+
### Yield Control
|
|
88
|
+
|
|
89
|
+
These parameters control how quickly and frequently the generator yields sentence fragments:
|
|
90
|
+
|
|
91
|
+
- `quick_yield_single_sentence_fragment: bool = False`
|
|
92
|
+
- When True, yields the first fragment of the first sentence as quickly as possible.
|
|
93
|
+
- Useful for getting immediate output in real-time applications like speech synthesis.
|
|
94
|
+
- Default: False
|
|
95
|
+
|
|
96
|
+
- `quick_yield_for_all_sentences: bool = False`
|
|
97
|
+
- When True, yields the first fragment of every sentence as quickly as possible.
|
|
98
|
+
- Extends the quick yield behavior to all sentences, not just the first one.
|
|
99
|
+
- Automatically sets `quick_yield_single_sentence_fragment` to True.
|
|
100
|
+
- Default: False
|
|
101
|
+
|
|
102
|
+
- `quick_yield_every_fragment: bool = False`
|
|
103
|
+
- When True, yields every fragment of every sentence as quickly as possible.
|
|
104
|
+
- Provides the most granular output, yielding fragments as soon as they're detected.
|
|
105
|
+
- Automatically sets both `quick_yield_for_all_sentences` and `quick_yield_single_sentence_fragment` to True.
|
|
106
|
+
- Default: False
|
|
107
|
+
|
|
108
|
+
### Text Cleanup
|
|
109
|
+
|
|
110
|
+
- `cleanup_text_links: bool = False`
|
|
111
|
+
- When True, removes hyperlinks from the output sentences.
|
|
112
|
+
- Default: False
|
|
113
|
+
|
|
114
|
+
- `cleanup_text_emojis: bool = False`
|
|
115
|
+
- When True, removes emoji characters from the output sentences.
|
|
116
|
+
- Default: False
|
|
117
|
+
|
|
118
|
+
### Tokenization
|
|
119
|
+
|
|
120
|
+
- `tokenize_sentences: Callable = None`
|
|
121
|
+
- Custom function for sentence tokenization.
|
|
122
|
+
- If None, uses the default tokenizer specified by `tokenizer`.
|
|
123
|
+
- Default: None
|
|
124
|
+
|
|
125
|
+
- `tokenizer: str = "nltk"`
|
|
126
|
+
- Specifies the tokenizer to use. Options: "nltk" or "stanza"
|
|
127
|
+
- Default: "nltk"
|
|
128
|
+
|
|
129
|
+
- `language: str = "en"`
|
|
130
|
+
- Language setting for the tokenizer.
|
|
131
|
+
- Use "en" for English or "multilingual" for Stanza tokenizer.
|
|
132
|
+
- Default: "en"
|
|
133
|
+
|
|
134
|
+
### Debugging and Fine-tuning
|
|
135
|
+
|
|
136
|
+
- `log_characters: bool = False`
|
|
137
|
+
- When True, logs each processed character to the console.
|
|
138
|
+
- Useful for debugging or monitoring real-time processing.
|
|
139
|
+
- Default: False
|
|
140
|
+
|
|
141
|
+
- `sentence_fragment_delimiters: str = ".?!;:,\n…)]}。-"`
|
|
142
|
+
- Characters considered as potential sentence fragment delimiters.
|
|
143
|
+
- Used for quick yielding of sentence fragments.
|
|
144
|
+
- Default: ".?!;:,\n…)]}。-"
|
|
145
|
+
|
|
146
|
+
- `full_sentence_delimiters: str = ".?!\n…。"`
|
|
147
|
+
- Characters considered as full sentence delimiters.
|
|
148
|
+
- Used for more definitive sentence boundary detection.
|
|
149
|
+
- Default: ".?!\n…。"
|
|
150
|
+
|
|
151
|
+
- `force_first_fragment_after_words: int = 15`
|
|
152
|
+
- Forces the yield of the first sentence fragment after this many words.
|
|
153
|
+
- Ensures timely output even with long opening sentences.
|
|
154
|
+
- Default: 15 words
|
|
155
|
+
|
|
156
|
+
## Contributing
|
|
157
|
+
|
|
158
|
+
Any Contributions you make are welcome and **greatly appreciated**.
|
|
159
|
+
|
|
160
|
+
1. **Fork** the Project.
|
|
161
|
+
2. **Create** your Feature Branch (`git checkout -b feature/AmazingFeature`).
|
|
162
|
+
3. **Commit** your Changes (`git commit -m 'Add some AmazingFeature'`).
|
|
163
|
+
4. **Push** to the Branch (`git push origin feature/AmazingFeature`).
|
|
164
|
+
5. **Open** a Pull Request.
|
|
165
|
+
|
|
166
|
+
## License
|
|
167
|
+
|
|
168
|
+
This project is licensed under the MIT License. For more details, see the [`LICENSE`](LICENSE) file.
|
|
169
|
+
|
|
170
|
+
---
|
|
171
|
+
|
|
172
|
+
Project created and maintained by [Kolja Beigel](https://github.com/KoljaB).
|
|
@@ -5,7 +5,7 @@ with open("README.md", "r") as fh:
|
|
|
5
5
|
|
|
6
6
|
setuptools.setup(
|
|
7
7
|
name="stream2sentence",
|
|
8
|
-
version="0.2.
|
|
8
|
+
version="0.2.5",
|
|
9
9
|
author="Kolja Beigel",
|
|
10
10
|
author_email="kolja.beigel@web.de",
|
|
11
11
|
description="Real-time processing and delivery of sentences from a continuous stream of characters or text chunks.",
|