normt2s 0.1.0__tar.gz

normt2s-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Christian Siegel
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

normt2s-0.1.0/PKG-INFO

@@ -0,0 +1,392 @@
+Metadata-Version: 2.4
+Name: normt2s
+Version: 0.1.0
+Summary: Lightweight, dependency-free text normalization (numbers, units, abbreviations) for TTS.
+Author: Christian Siegel
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/c-siegel/normt2s
+Project-URL: Repository, https://github.com/c-siegel/normt2s
+Project-URL: Issues, https://github.com/c-siegel/normt2s/issues
+Keywords: tts,text-normalization,num2words,speech
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Text Processing :: Linguistic
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Dynamic: license-file
+
+# normt2s
+
+**Make written text speakable.** `normt2s` turns numbers, units, abbreviations, dates and math into
+the words a text-to-speech engine should say, so `3.5 km/h` is not read as "three dot five k m
+slash h".
+
+- **English and German**
+- **No dependencies**, pure Python 3.9+, MIT licensed
+- **Fast**: about 90 µs for a typical sentence
+- **Streaming-ready**: works on LLM output as it arrives, token by token
+- **Easy to extend**: add words with one line of code or a small JSON file
+
+```python
+from normt2s import normalize
+
+normalize("It costs $3.50 for 2 kg, e.g. in 1984.")
+# 'It costs three dollars and fifty cents for two kilograms, for example in nineteen eighty-four.'
+
+normalize("Der Zug fährt 120 km/h, z. B. um 14:30.", lang="de")
+# 'Der Zug fährt einhundertzwanzig Kilometer pro Stunde, zum Beispiel um vierzehn Uhr dreißig.'
+```
+
+## Contents
+
+1. [Install](#install)
+2. [What gets converted](#what-gets-converted)
+3. [Adding your own words](#adding-your-own-words)
+4. [Math (LaTeX)](#math-latex)
+5. [Streaming LLM output](#streaming-llm-output)
+6. [Reference: options and API](#reference-options-and-api)
+7. [How it works](#how-it-works)
+8. [Adding a language](#adding-a-language)
+9. [Performance](#performance)
+10. [Known limitations](#known-limitations)
+11. [Development](#development)
+
+## Install
+
+`normt2s` is available on [PyPI](https://pypi.org/project/normt2s/):
+
+```
+pip install normt2s
+```
+
+To work on the library itself, install from a clone of the repository instead:
+
+```
+pip install -e .
+```
+
+## What gets converted
+
+Every row is real output from the library.
+
+### English
+
+| Category | Input | Spoken |
+|---|---|---|
+| Numbers | `1,234 points` | one thousand two hundred thirty-four points |
+| | `3.14`, `-5`, `007` | three point one four, minus five, zero zero seven |
+| | `1st`, `23rd` | first, twenty-third |
+| | `50%`, `1/2 cup` | fifty percent, one half cup |
+| Years | `in 1984`, `the 1990s` | in nineteen eighty-four, the nineteen nineties |
+| | `I have 1984 apples` | I have one thousand nine hundred eighty-four apples |
+| Time | `3:45 pm`, `at 14:30` | three forty-five p m, at fourteen thirty |
+| Dates | `March 3rd, 2024`, `2024-03-15` | March third, twenty twenty-four; March fifteenth, twenty twenty-four |
+| Units | `100 km/h`, `20°C`, `9.81 m/s²`, `1 ft` / `2 ft` | one hundred kilometers per hour, twenty degrees Celsius, nine point eight one meters per second squared, one foot / two feet |
+| Currency | `$3.50`, `€5`, `5 EUR`, `$2.5 million` | three dollars and fifty cents, five euros, five euros, two point five million dollars |
+| Abbreviations | `Dr. Smith, e.g. ...`, `etc.` | Doctor Smith, for example ..., et cetera |
+| Math symbols | `2 + 3 = 5`, `x²`, `a ≤ b` | two plus three equals five, x squared, a is less than or equal to b |
+
+### German
+
+| Category | Input | Spoken |
+|---|---|---|
+| Numbers | `1.234,5` | eintausendzweihundertvierunddreißig Komma fünf |
+| | `1/3`, `3/4` | ein drittel, drei viertel |
+| Dates | `15.03.2024` | fünfzehnter März zweitausendvierundzwanzig |
+| | `am 15.03.2024` | am fünfzehnten März zweitausendvierundzwanzig |
+| | `am 3. Mai`, `im Mai 2024` | am dritten Mai, im Mai zweitausendvierundzwanzig |
+| Time | `um 14:30` | um vierzehn Uhr dreißig |
+| Units | `1 kg`, `1 min`, `21 min` | ein Kilogramm, eine Minute, einundzwanzig Minuten |
+| Currency | `3,50 €`, `5 €` | drei Euro fünfzig Cent, fünf Euro |
+| Abbreviations | `z. B.`, `ca. 5 Mio.` | zum Beispiel, circa fünf Millionen |
+
+German dates take the right case ending from the word before them: `am` / `vom` / `zum` / `dem`
+/ `den` give *fünfzehnten*, `der` gives *fünfzehnte*, and a date on its own gives *fünfzehnter*.
+
+### What is left alone
+
+Code (`` `inline` `` and fenced blocks), URLs, e-mail addresses, HTML tags and Markdown link
+targets are never changed:
+
+```python
+normalize("Run `pip install numpy==1.26` and open https://example.com/v1.2/page")
+# unchanged
+```
+
+Things that only look like numbers are kept too: versions with several dots are read digit by
+digit (`3.2.1` is "three point two point one"), and letters glued to digits (`H2O`, `mp3`) are
+not split.
+
+## Adding your own words
+
+There is one rule to remember: **create a `Normalizer`, add words to it, and use that instance.**
+
+```python
+from normt2s import Normalizer, normalize
+
+n = Normalizer("en")
+n.add_abbreviation("LOLA", "Lola")
+n.add_unit("kn", "knot")                                      # plural defaults to "knots"
+n.add_unit("ch", "child", "children")                         # or give the plural yourself
+n.add_currency("CHF", ("franc", "francs"), ("rappen", "rappen"))
+
+n.normalize("LOLA sails 12 kn with 2 ch for 5.20 CHF")
+# 'Lola sails twelve knots with two children for five francs and twenty rappen'
+
+normalize("LOLA sails 12 kn", n)       # the helper also accepts your instance
+```
+
+> **Gotcha:** `normalize(text, "en")` uses a separate, shared instance. Words you add to your own
+> `Normalizer("en")` are **not** visible there. Pass your instance (`normalize(text, n)`), or edit
+> the shared one with `get_normalizer("en")`.
+
+Your entries override the built-in ones, and changes apply immediately, even to an instance you
+have already used.
+
+### Which method for what
+
+| You want to change | Method | Example |
+|---|---|---|
+| An abbreviation | `add_abbreviation(key, to)` | `("tbh", "to be honest")` |
+| A unit after a number | `add_unit(symbol, singular, plural, gender)` | `("kn", "knot")` |
+| A currency | `add_currency(symbol, major, minor)` | see above |
+| A math symbol such as `=` or `×` | `add_symbol(key, word)` | `("=", "is")` |
+| A LaTeX command | `lexicon.add_latex_command(name, spoken)` | `("R", "the reals")` |
+| A LaTeX function | `lexicon.add_latex_function(name, spoken)` | `("sinc", "sinc")` |
+| A LaTeX sentence template | `lexicon.add_phrase(key, template)` | `("frac", "{a} divided by {b}")` |
+| Remove an entry | `lexicon.remove(kind, key)` | `("abbreviations", "vs.")` |
+
+Notes on abbreviations:
+
+- Matching is case sensitive, but for lowercase keys a capitalised variant is added for you
+  (`e.g.` also gives `E.g.` → "For example"). Pass `capitalized=False` to turn that off.
+- Spaces inside a key are optional when matching: `z. B.` also matches `z.B.`
+- Set `final=True` for abbreviations that can end a sentence (`etc.`), so the full stop is kept
+  when a new sentence follows.
+
+### Using a file
+
+Put your words in a JSON (or, on Python 3.11+, TOML) file and load it:
+
+```python
+n.load_lexicon("my_terms.json")
+```
+
+```json
+{
+  "abbreviations": {
+    "LMS": "learning management system",
+    "etc.": {"to": "et cetera", "final": true}
+  },
+  "units": {"kn": ["knot"], "ch": ["child", "children"]},
+  "currencies": {"CHF": {"major": ["franc", "francs"], "minor": ["rappen", "rappen"]}},
+  "symbols": {"=": "equals"},
+  "latex": {
+    "commands": {"R": "the reals"},
+    "functions": {"sinc": "sinc"},
+    "phrases": {"frac": "{a} divided by {b}"}
+  }
+}
+```
+
+The built-in data uses exactly this format: see [src/normt2s/data/en.json](src/normt2s/data/en.json)
+and [de.json](src/normt2s/data/de.json). A unit entry is `[singular, plural, gender]`, where the
+plural and gender are optional and the gender (`"m"`, `"f"`, `"n"`) only matters in German.
+
+## Math (LaTeX)
+
+Math in `$...$`, `$$...$$`, `\(...\)` and `\[...\]` is read aloud:
+
+```python
+normalize(r"The area is $\pi r^2$.")
+# 'The area is pi r squared.'
+
+normalize(r"Solve $\frac{a+b}{c} = \sqrt{x}$ now")
+# 'Solve the fraction a plus b over c, end fraction equals square root of x now'
+
+normalize(r"Es gilt $x \in \mathbb{R}$ und $\sum_{i=1}^{n} i$.", lang="de")
+# 'Es gilt x ist Element von den reellen Zahlen und die Summe von i gleich eins bis n über i.'
+```
+
+Supported: fractions, roots, powers and subscripts, sums, products, integrals and limits, Greek
+letters, relations and set notation, functions such as `\sin` and `\log`, absolute values and
+norms, accents (`\vec`, `\hat`), `\binom`, factorials, primes, matrices, `cases`, and `\text{}` /
+`\mathrm{}`. Units after a number work too: `5\,\mathrm{km}` is "five kilometers".
+
+Good to know:
+
+- **Parentheses** are only spoken where they change the meaning: `(a+b)^2` and `2(x+1)` get
+  "open paren ... close paren", `y = (x+1)` does not.
+- **Complex arguments** are fenced so the listener can follow them: "the square root of x plus
+  one, end root".
+- **Malformed LaTeX** never raises. It is left exactly as it was written.
+- **Dollar signs:** a single `$` only starts math under the pandoc rule, so
+  `from $5 to $6` stays currency.
+- **Turn it off** with `Normalizer("en", latex="keep")` to leave all math untouched.
+- **Commands vs. functions:** `add_latex_command` is for symbol-like names (`\cdot`, `\R`),
+  `add_latex_function` for names applied to an argument (`\sinc x`). Commands win when both exist,
+  and `add_latex_function` warns if a command with that name already exists.
+
+## Streaming LLM output
+
+When the LLM answers token by token, you do not want to wait for the full answer, and you cannot
+normalize each token alone (`"1"`, `"20"`, `" km"` would be read separately). The stream wrapper
+buffers text and releases it only when it is safe:
+
+```python
+stream = n.stream()
+for token in llm_tokens:
+    speakable = stream.feed(token)     # "" until a safe boundary is reached
+    if speakable:
+        tts.say(speakable)
+tts.say(stream.flush())                # call once at the end
+```
+
+or, as a generator:
+
+```python
+for speakable in n.normalize_stream(llm_tokens):
+    tts.say(speakable)
+```
+
+What that looks like:
+
+```
+feed 'The train '   -> ''
+feed 'goes 1'       -> ''
+feed '20 km'        -> ''
+feed '/h. Dr'       -> 'The train goes one hundred twenty kilometers per hour. '
+feed '. Smith '     -> ''
+feed 'pays $3'      -> ''
+feed '.50 now'      -> ''
+feed '. Done'       -> ''
+flush               -> 'Doctor Smith pays three dollars and fifty cents now. Done'
+```
+
+Text is released at the end of a sentence or at a newline. It is never cut inside a number and its
+unit, an abbreviation (`Dr. Smith`), a date (`am 3. Mai`), a code block or a math span. The
+combined output is identical to normalizing the whole text at once, whatever the chunk sizes (this
+is tested). If a long run has no boundary, it is flushed after `max_buffer` characters (default
+400) at a whitespace that does not follow a number.
+
+The trade-off: speech starts at the end of the first sentence, not at the first token.
+
+## Reference: options and API
+
+```python
+Normalizer(lang="en", lexicon=None, latex="speak")
+```
+
+| Argument | Meaning |
+|---|---|
+| `lang` | `"en"` or `"de"` (or a `Language` object) |
+| `lexicon` | a `Lexicon` to use instead of the built-in data |
+| `latex` | `"speak"` (default) reads math aloud, `"keep"` leaves it untouched |
+
+| Call | What it does |
+|---|---|
+| `normalize(text, lang="en")` | one-shot helper; `lang` may be a code or your own `Normalizer` |
+| `get_normalizer(lang)` | the shared instance behind `normalize(text, "en")` |
+| `n.normalize(text)` (or `n(text)`) | normalize a string |
+| `n.stream(max_buffer=400)` | a `StreamNormalizer` with `feed()` and `flush()` |
+| `n.normalize_stream(chunks)` | generator over speakable pieces |
+| `n.add_*`, `n.load_lexicon(path)` | extend the dictionaries (see above) |
+| `n.lexicon` | the underlying `Lexicon` with the LaTeX methods |
+
+## How it works
+
+Text is split into spans. Code, URLs, e-mails and HTML are kept as they are, math goes to the
+LaTeX speaker, and everything else runs through these steps:
+
+1. **Abbreviations** (`Dr.` → "Doctor")
+2. **Dates** (`15.03.2024`, `March 3rd`)
+3. **Symbols that need digits** (`5 - 3`, `x^2`)
+4. **Numbers**, together with what follows them: units, percent, currency, ordinals, am/pm
+5. **Remaining symbols** (`=`, `+`, `²`, `≤`)
+
+All regular expressions are compiled once and rebuilt only when the dictionaries change.
+
+```
+src/normt2s/
+  __init__.py      public API: normalize, get_normalizer
+  normalizer.py    the pipeline above (start reading here)
+  lexicon.py       the editable dictionaries and how they are loaded
+  stream.py        buffering for streamed text
+  latex.py         LaTeX tokenizer, parser and speaker
+  langs/           per-language rules: number words, times, dates, fractions
+  data/            en.json, de.json: abbreviations, units, currencies, symbols, math vocabulary
+tests/             examples of every rule, a good way to see expected behaviour
+benchmarks/        bench.py
+```
+
+Rules that need code (how to build "twenty-one") live in `langs/`. Everything that is just a word
+list lives in `data/`.
+
+## Adding a language
+
+1. Create `src/normt2s/langs/xx.py` with a subclass of `Language`: set the separators and words
+   (`thousands_sep`, `decimal_sep`, `minus`, `point`, `percent`, ...) and implement `cardinal`,
+   `ordinal`, `time` and `fraction`. Optionally add `year`, `decade`, `date_regex` and
+   `format_date`. End the file with `register(YourLanguage())`.
+2. Import it in `langs/__init__.py`.
+3. Create `src/normt2s/data/xx.json` in the format shown above.
+
+`langs/en.py` and `langs/de.py` are working examples.
+
+## Performance
+
+`python benchmarks/bench.py` on a laptop:
+
+| Input | Time |
+|---|---|
+| a typical sentence (74 characters) | about 90 µs |
+| a 600-character paragraph | about 0.9 ms |
+| one-time startup (building and compiling) | about 30 ms |
+
+## Known limitations
+
+- **Acronyms** such as "AI" or "LLM" are not expanded by default. Add the ones you need with
+  `add_abbreviation`.
+- **Subtraction:** `-` is only read as "minus" between two numbers with spaces around it
+  (`5 - 3`). In `x - 1` it is left as it is. In LaTeX math it is always read.
+- **German ordinal dots** are only recognised in dates (`am 3. Mai`), not in general
+  (`der 3. Platz`).
+- **German number + noun:** `1 Stunde` reads "eins Stunde", because the gender of a spelled-out
+  noun is unknown. Use the unit symbol (`1 h`) to get "eine Stunde".
+- **Dates:** English `3/15/2024` and `15 March 2024` are not handled, and neither are German
+  two-digit years.
+- **Durations** such as `1:30:15` are not recognised.
+- **Single capital-letter units** (`V`, `W`, `L`) only match when attached to the number (`12V`),
+  so that "5 A grade" is not read as five amperes.
+- **LaTeX:** a lone `|` reads as "such that", so `P(A|B)` is not "given". Layout commands such as
+  `\overset` and `\underbrace` are not interpreted.
+
+## Development
+
+```
+pip install -e .[dev]
+pytest
+python benchmarks/bench.py
+```
+
+The tests double as documentation: `tests/test_en.py` and `tests/test_de.py` list the expected
+output for each rule.
+
+## AI notice
+
+This package was co-authored with Claude Code (Claude Sonnet 5.5, Anthropic). The code, tests and
+documentation were written in collaboration between a human maintainer and the AI assistant.
+Please review the output against your own requirements before relying on it in production.
+
+## License
+
+MIT, see [LICENSE](LICENSE).