phonemizer_ffi 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA256:
3
+ metadata.gz: 3fa8606189dfc434db732488a0ec2432bc7b461bff6c8292505f57b381a79b52
4
+ data.tar.gz: e38b5465a4a9030d3e0e55e43d1c2bf701a4b1fc49d866dedc0edf7b933dd084
5
+ SHA512:
6
+ metadata.gz: e9f82f4f3d809ee1c0321edd6c8fbf608fd81db2db47cdfa0d72497b6472f388773a0c30fbcc04296bece3978d33f44648ea93a2e5c2610fa7c9d4ef10836bae
7
+ data.tar.gz: 4ebac7f564269579616792484f35ca484178e768dee0830f423690a3e89485bf187a2a896a36f0dd13236b4ac2ee2685bdb4b6f46b32c5e90b63768661fddc4f
data/CHANGELOG.md ADDED
@@ -0,0 +1,9 @@
1
+ # Changelog
2
+
3
+ ## 0.1.0 - 2026-11-22
4
+
5
+ - Initial release. `lib/phonemizer_ffi.rb` (extracted from the `scansion` gem) packaged as a standalone gem: a drop-in, API-compatible replacement for the `phonemizer` gem that calls libespeak-ng in-process via FFI instead of fork+exec-ing the espeak-ng CLI per word (~40x faster on a 500-word batch, byte-identical output in default mode)
6
+ - `phonemize_ffi` executable (`bin/phonemize_ffi`), registered via `spec.executables` and routed through the new `Phonemizer.main(argv)` entry point; adds a stdin batch mode (one utterance per line) since the FFI backend has no worker pool; the lib file also still runs standalone (`ruby lib/phonemizer_ffi.rb ...`)
7
+ - Runtime dependency on the `ffi` gem; requires the espeak-ng shared library and data (`apt install libespeak-ng-dev`, or `PHONEMIZER_ESPEAK_LIB` for non-standard lib paths such as keg-only Homebrew installs)
8
+ - RSpec suite under `spec/`: deterministic unit specs (no espeak needed) + integration specs skipped when libespeak-ng is unavailable
9
+ - Documentation and comments in English
data/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 anantone
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
data/README.md ADDED
@@ -0,0 +1,268 @@
1
+ phonemizer_ffi
2
+ ==============
3
+
4
+ [![Gem Version](https://img.shields.io/gem/v/phonemizer_ffi.svg)](https://rubygems.org/gems/phonemizer_ffi)
5
+ [![MIT License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
6
+
7
+ An in-process [espeak-ng](https://github.com/espeak-ng/espeak-ng) phonemizer for Ruby: a **drop-in replacement for the [`phonemizer`](https://rubygems.org/gems/phonemizer) gem** that calls the `libespeak-ng` C library directly via [FFI](https://rubygems.org/gems/ffi) instead of spawning the `espeak-ng` CLI process once per word. Same API, same output — typically **one to two orders of magnitude faster** on large batches. Supports [these languages](https://github.com/espeak-ng/espeak-ng/blob/master/docs/languages.md).
8
+
9
+
10
+ Why phonemizer_ffi instead of phonemizer?
11
+ ----------------------------------------------------------------
12
+
13
+ | | `phonemizer` | `phonemizer_ffi` |
14
+ |---|---|---|
15
+ | Backend | `espeak-ng` CLI, one fork+exec per word | `libespeak-ng` via FFI, plain in-process function call |
16
+ | Process spawns | one per word | none |
17
+ | Parallelism | worker pool (`workers:`, default CPUs − 1) | none needed — calls are that cheap; `workers:` is accepted but ignored |
18
+ | Extra install | `espeak-ng` binary only | `ffi` gem + `libespeak-ng` shared library and data (`libespeak-ng-dev`) |
19
+ | API | `Phonemizer.phonemize(...)` | **identical** (drop-in: swap the `require`) |
20
+
21
+ Measured on this machine (Ruby 4.0, espeak-ng 1.52, 500 unique French words from Lexique):
22
+
23
+ ```text
24
+ phonemizer (CLI, fork+exec per word): 0.79 s
25
+ phonemizer_ffi (FFI, in-process): 0.02 s # ~40x faster
26
+ ```
27
+
28
+ The CLI wrapper pays a fork+exec+pipe round-trip per word (~1–2 ms each); the FFI backend pays a single in-process function call (microseconds). Process-spawn overhead — not phonemization itself — dominates the CLI gem's runtime, so on a 250k-word lexicon the difference is minutes vs. seconds.
29
+
30
+ **Output parity:** on the same 500-word sample, `phonemizer_ffi` produces byte-identical output to `phonemizer` in the default contiguous mode (`phone_sep: ""`). In sep mode the only possible difference is a word boundary being spaced once instead of twice around liaison hyphens — an inherent espeak-ng `--sep` limitation, documented in the comments of `lib/phonemizer_ffi.rb`.
31
+
32
+ Trade-offs:
33
+
34
+ - You need the espeak-ng **shared library and data**, not just the CLI binary (`apt install libespeak-ng-dev`; on macOS the Homebrew lib is keg-only — set `PHONEMIZER_ESPEAK_LIB`, see below).
35
+ - Like the CLI gem, this gem is **not thread-safe** (libespeak-ng keeps global voice state); wrap calls in your own `Mutex` if you call from several threads. There is no worker pool to tune — one process is usually faster than the CLI's parallel spawns anyway.
36
+ - Do not load both gems in the same process: both define the `Phonemizer` module on purpose (drop-in replacement) and their constants would clash.
37
+
38
+
39
+ Requirements
40
+ ----------------------------------------------------------------
41
+
42
+ - Ruby 3.1+
43
+ - The `ffi` gem (declared as a runtime dependency of this gem)
44
+ - The **espeak-ng shared library** and its data:
45
+ ```bash
46
+ apt install libespeak-ng-dev # Debian/Ubuntu (lib + espeak-ng-data)
47
+ brew install espeak-ng # macOS
48
+ ```
49
+ If the library lives outside the default search path (e.g. a keg-only Homebrew install), point `PHONEMIZER_ESPEAK_LIB` at it before loading the gem:
50
+ ```ruby
51
+ ENV["PHONEMIZER_ESPEAK_LIB"] = %x(brew --prefix espeak-ng).strip + "/lib/libespeak-ng.dylib"
52
+ ```
53
+
54
+
55
+ Installation
56
+ ----------------------------------------------------------------
57
+
58
+ Add to your Gemfile:
59
+
60
+ ```ruby
61
+ gem "phonemizer_ffi"
62
+ ```
63
+
64
+ Or install directly:
65
+
66
+ ```bash
67
+ gem install phonemizer_ffi
68
+ ```
69
+
70
+
71
+ Usage
72
+ ----------------------------------------------------------------
73
+
74
+ Same API as the `phonemizer` gem — swap `require "phonemizer"` for `require "phonemizer_ffi"`:
75
+
76
+ ```ruby
77
+ require "phonemizer_ffi"
78
+
79
+ # Single word
80
+ Phonemizer.phonemize("manger", language: "fr-fr")
81
+ # => "mɑ̃ʒe"
82
+
83
+ # Batch - pass an Array, get an Array back (same order). This is where the
84
+ # speedup shows: one in-process pass, no process spawned per word.
85
+ Phonemizer.phonemize(["docteur", "tableau", "pastille"], language: "fr-fr")
86
+ # => ["dɔktœʁ", "tablo", "pastij"]
87
+
88
+ # A whole lexicon in one call (the internal cache dedupes repeated forms)
89
+ lines = File.readlines("lexique.txt", chomp: true).reject(&:empty?)
90
+ ipa = Phonemizer.phonemize(lines, language: "fr-fr") # Array of IPA strings, same order
91
+ ```
92
+
93
+
94
+ Options
95
+ ----------------------------------------------------------------
96
+
97
+ ```ruby
98
+ Phonemizer.phonemize(
99
+ "les amis",
100
+ language: "fr-fr", # espeak voice (default "en-us")
101
+ phone_sep: "", # separator between phones ("" = contiguous)
102
+ word_sep: " ", # separator between words in one utterance
103
+ strip_stress: true, # remove stress marks and liaison hyphens
104
+ language_switch: :remove, # :remove or :keep, for inline (fr)/(en) tags
105
+ workers: nil # accepted for drop-in compatibility, IGNORED:
106
+ # the FFI backend is single-threaded in-process
107
+ )
108
+ ```
109
+
110
+ Separate each phone explicitly:
111
+
112
+ ```ruby
113
+ Phonemizer.phonemize("tableau", language: "fr-fr", phone_sep: " ")
114
+ # => "t a b l o"
115
+ ```
116
+
117
+ Keep the language-switch markers espeak emits mid-utterance:
118
+
119
+ ```ruby
120
+ Phonemizer.phonemize("le célèbre Schwarzenegger", language: "fr-fr", language_switch: :keep)
121
+ # => "lə selɛbʁ (en)ʃwɔːzənɛɡə(fr)"
122
+ ```
123
+
124
+
125
+ Command line
126
+ ----------------------------------------------------------------
127
+
128
+ The `phonemize_ffi` executable exposes the same options as `phonemize`, minus `-j/--workers`, plus a **stdin batch mode** (one utterance per line — the natural high-throughput path for this backend, since there is no worker pool). Pass text as arguments, or a file path to phonemize it line by line:
129
+
130
+ ```bash
131
+ phonemize_ffi "hello world"
132
+ phonemize_ffi -l fr-fr "les amis"
133
+ phonemize_ffi -l fr-fr words.txt > words.ipa
134
+ cat words.txt | phonemize_ffi -l fr-fr > words.ipa # stdin batch mode
135
+ ```
136
+
137
+ ```bash
138
+ phonemize_ffi --help
139
+ # Usage: phonemize_ffi [options] [text | file]
140
+ # -l, --language LANG espeak voice (default: en-us)
141
+ # -p, --phone-sep SEP separator between phones (default: none)
142
+ # -w, --word-sep SEP separator between words (default: space)
143
+ # -s, --[no-]strip-stress strip stress marks and liaison hyphens (default: yes)
144
+ # -k, --keep-language-switch keep inline (en)/(fr) language tags (default: remove)
145
+ # -h, --help show this help
146
+ ```
147
+
148
+ ```bash
149
+ phonemize_ffi -l fr-fr -p " " tableau
150
+ # => t a b l o
151
+ ```
152
+
153
+ The library file also runs standalone (`ruby lib/phonemizer_ffi.rb ...`), so it can be vendored as a single file.
154
+
155
+
156
+ Punctuation
157
+ ----------------------------------------------------------------
158
+
159
+ Punctuation is preserved verbatim and never sent to espeak, so whole sentences pass through cleanly:
160
+
161
+ ```ruby
162
+ Phonemizer.phonemize("Bonjour, comment ça va ?", language: "fr-fr")
163
+ # => "bɔ̃ʒuʁ,kɔmɑ̃ sa va?"
164
+ ```
165
+
166
+ Apostrophes are kept inside the word on purpose, so elisions stay intact and liaison consonants aren't lost:
167
+
168
+ ```ruby
169
+ Phonemizer.phonemize("aujourd'hui", language: "fr-fr")
170
+ # => "oʒuʁdyi"
171
+ ```
172
+
173
+
174
+ Caching
175
+ ----------------------------------------------------------------
176
+
177
+ Results are cached per `(word, language, options)` — with the FFI backend every call is an in-process function call, and the cache makes repeated forms free. Useful when the same forms recur across a large lexicon. Call `clear_cache` to free memory or start fresh:
178
+
179
+ ```ruby
180
+ Phonemizer.clear_cache
181
+ ```
182
+
183
+
184
+ How it works
185
+ ----------------------------------------------------------------
186
+
187
+ - `espeak_TextToPhonemes` transcribes one *clause* per call (text up to a comma, semicolon, colon, period…), advancing its text pointer. The pipeline splits punctuation out first, so each clean piece is transcribed on its own — adjacent pieces are never merged into one buffer, because the C API gives no isolation between concatenated chunks (unlike the CLI's `--stdin`, newlines are just spaces there).
188
+ - The pipeline always asks espeak for the `\x01` sentinel byte as inter-phone separator (bits 8-23 of `phonememode`, the same field as `--sep`). `\x01` never occurs in IPA output, so any user `phone_sep` — multi-character, containing spaces, even `"-"` — is substituted afterwards in a single pass without collision risk. `espeak_raw` instead passes single-byte separators straight through, for byte-level parity with the CLI's `--sep`.
189
+ - Voice selection mirrors the CLI: `espeak_SetVoiceByName` for installed voices/variants, falling back to `espeak_SetVoiceByProperties` on the language code (e.g. `"fr-fr"` when only `"fr"` is installed).
190
+
191
+
192
+ Thread Safety
193
+ ----------------------------------------------------------------
194
+
195
+ This gem is **not thread-safe**: libespeak-ng keeps global voice state and the internal cache is a plain Hash without synchronization. If you call `phonemize` from multiple threads, wrap your calls in your own `Mutex`:
196
+
197
+ ```ruby
198
+ @mutex = Mutex.new
199
+ @mutex.synchronize { Phonemizer.phonemize(words, language: "fr-fr") }
200
+ ```
201
+
202
+
203
+ API Reference
204
+ ----------------------------------------------------------------
205
+
206
+ ### `Phonemizer.phonemize(words, **options)`
207
+
208
+ Phonemize text to IPA. Returns a String for String input, Array for Array input.
209
+
210
+ **Arguments:**
211
+ - `words` — String or Array of Strings
212
+ - `language` — espeak voice (default: `"en-us"`)
213
+ - `phone_sep` — separator between phones (default: `""`)
214
+ - `word_sep` — separator between words in multi-word input (default: `" "`)
215
+ - `strip_stress` — remove stress marks ˈ ˌ and liaison hyphens (default: `true`)
216
+ - `language_switch` — `:remove` or `:keep` for inline language tags (default: `:remove`)
217
+ - `workers` — accepted for drop-in compatibility with the `phonemizer` gem, **ignored** (in-process, single-threaded backend)
218
+
219
+ ### `Phonemizer.espeak_raw(text, language, phone_sep: nil)`
220
+
221
+ Low-level: transcribe one utterance via libespeak-ng directly. Returns the raw IPA output string (stress marks, liaison hyphens and language tags intact).
222
+
223
+ **Arguments:**
224
+ - `text` — text to phonemize
225
+ - `language` — espeak voice
226
+ - `phone_sep` — optional separator (single-byte separators are passed to espeak as-is, like `--sep`; multi-byte ones fall back to the sentinel and are substituted)
227
+
228
+ ### `Phonemizer.clear_cache`
229
+
230
+ Clear the internal phonemization cache. Returns the new empty cache Hash.
231
+
232
+ ### `Phonemizer.main(argv = ARGV)`
233
+
234
+ Command-line entry point used by the `phonemize_ffi` executable.
235
+
236
+
237
+ Development
238
+ ----------------------------------------------------------------
239
+
240
+ ```bash
241
+ bundle install
242
+ bundle exec rspec # or: rake
243
+ ```
244
+
245
+ The integration specs are skipped automatically when libespeak-ng cannot be loaded in-process; the unit specs are deterministic and need no espeak at all.
246
+
247
+
248
+ Structure
249
+ ----------------------------------------------------------------
250
+
251
+ ```
252
+ ├── bin/phonemize_ffi executable (thin wrapper around Phonemizer.main)
253
+ ├── lib/phonemizer_ffi.rb library + CLI entry point Phonemizer.main
254
+ ├── lib/phonemizer_ffi/version.rb Phonemizer::VERSION
255
+ └── spec/ RSpec suite (unit + integration)
256
+ ```
257
+
258
+
259
+ Contributing
260
+ ----------------------------------------------------------------
261
+
262
+ Bug reports and pull requests are welcome on [GitHub](https://github.com/anantone/phonemizer_ffi).
263
+
264
+
265
+ License
266
+ ----------------------------------------------------------------
267
+
268
+ MIT
data/bin/phonemize_ffi ADDED
@@ -0,0 +1,11 @@
1
+ #!/usr/bin/env ruby
2
+ # frozen_string_literal: true
3
+
4
+ # Allows running the executable straight from a working copy (checkout); when
5
+ # the gem is installed, lib is already in the load path and this is a no-op.
6
+ lib = File.expand_path("../lib", __dir__)
7
+ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
8
+
9
+ require "phonemizer_ffi"
10
+
11
+ Phonemizer.main(ARGV)
@@ -0,0 +1,5 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Phonemizer
4
+ VERSION = '0.1.0'
5
+ end
@@ -0,0 +1,391 @@
1
+ #!/usr/bin/env ruby
2
+ # frozen_string_literal: true
3
+
4
+ require "ffi"
5
+ require_relative "phonemizer_ffi/version"
6
+
7
+ # Drop-in replacement for the espeak-ng CLI wrapper (the `phonemizer` gem),
8
+ # calling the espeak-ng C library directly via FFI — no process spawned per
9
+ # word.
10
+ #
11
+ # apt install libespeak-ng-dev # Debian/Ubuntu
12
+ # brew install espeak-ng # macOS (the lib is keg-only, see below)
13
+ #
14
+ # If the library lives outside the default search path (e.g. a keg-only
15
+ # Homebrew install), set PHONEMIZER_ESPEAK_LIB in the environment before
16
+ # loading this file:
17
+ #
18
+ # ENV["PHONEMIZER_ESPEAK_LIB"] =
19
+ # %x(brew --prefix espeak-ng).strip + "/lib/libespeak-ng.dylib"
20
+ #
21
+ # Backend contract (verified against libespeak-ng 1.1.51):
22
+ # espeak_TextToPhonemes translates one *clause* per call — the text up to
23
+ # a comma, semicolon, colon, period or similar terminator — while advancing
24
+ # its text pointer, and returns NULL at end of input. Words within the same
25
+ # clause influence each other (liaisons, pronounced final consonants).
26
+ #
27
+ # IMPORTANT difference from the CLI binary: the function offers NO isolation
28
+ # between concatenated chunks — newlines are just spaces and adjacent chunks
29
+ # merge. Grouping several words into one buffer would therefore silently
30
+ # corrupt the results; each piece gets its own call. It is still an
31
+ # in-process function call, far cheaper than the CLI's fork+exec per word.
32
+ #
33
+ # --ipa corresponds to bit 1 of phonememode (0x02); --sep to bits 8-23. The
34
+ # phonemize pipeline always asks espeak for the \x01 sentinel byte as the
35
+ # inter-phone separator, so any user phone_sep (multi-character, spaces,
36
+ # "-") can be substituted on the output without collision risk; espeak_raw,
37
+ # on the other hand, passes single-byte separators through as-is, like --sep.
38
+ module Phonemizer
39
+ module_function
40
+
41
+ DEFAULT_LANGUAGE = "en-us"
42
+
43
+ # Kept for API compatibility with the CLI wrapper. In-process calls are
44
+ # cheap and libespeak-ng is not thread-safe: there is nothing to pool.
45
+ DEFAULT_WORKERS = 1
46
+
47
+ # espeak emits parenthesized language tags on language switches, e.g.
48
+ # (en) / (fr) / (pt-br); the pattern covers any tag espeak may emit.
49
+ LANG_SWITCH = /\((?:[a-z]{2,3}(?:-[a-z0-9]+)*)\)/
50
+
51
+ # Characters espeak would try to pronounce; we split them out instead.
52
+ # Apostrophes are EXCLUDED on purpose: they must stay inside the word they
53
+ # elide (aujourd'hui, l'instar). Cutting them would transcribe the two
54
+ # halves separately and lose the liaison consonants (oʒuʁdɥi -> oʒuʁ + yi).
55
+ PUNCT = /([[:punct:]&&[^'’‘]]+)/
56
+
57
+ # Cache shared by the module functions. A constant (not an @ivar) so the
58
+ # methods behave the same whether called as Phonemizer.phonemize(...) or
59
+ # after `include Phonemizer` — module_function copies don't carry the
60
+ # module's instance-variable context.
61
+ CACHE = {}
62
+
63
+ # ---------------------------------------------------------------- FFI
64
+
65
+ module LibEspeak
66
+ extend FFI::Library
67
+
68
+ ffi_lib [ENV["PHONEMIZER_ESPEAK_LIB"], "libespeak-ng.so.1",
69
+ "libespeak-ng.so", "libespeak-ng.dylib"].compact
70
+
71
+ AUDIO_OUTPUT_SYNCHRONOUS = 2 # no audio output, no async queue
72
+ CHARS_UTF8 = 1 # espeakCHARS_UTF8
73
+ INITIALIZE_DONT_EXIT = 0x8000 # return an error value instead of exit()
74
+ PHONEME_IPA = 0x02 # bit 1 of phonememode: IPA output
75
+
76
+ EE_OK = 0
77
+ EE_INTERNAL_ERROR = -1
78
+
79
+ # Sentinel inter-phone separator requested from espeak (encoded in bits
80
+ # 8-23 of phonememode). \x01 never appears in IPA output, so substituting
81
+ # it afterwards with the user's phone_sep cannot collide, whatever the
82
+ # phone_sep is (multi-character, spaces, even "-").
83
+ SEP_CHAR = "\x01"
84
+
85
+ attach_function :espeak_Initialize, %i[int int pointer int], :int
86
+ attach_function :espeak_SetVoiceByName, [:string], :int
87
+ attach_function :espeak_SetVoiceByProperties, [:pointer], :int
88
+ attach_function :espeak_TextToPhonemes, %i[pointer int int], :string
89
+
90
+ # espeak_VOICE — only the fields we fill in; the rest stay zero.
91
+ # NOTE: FFI forbids ASSIGNING :string struct fields inline (they are
92
+ # read-only shortcuts for char*). char* members are therefore declared
93
+ # :pointer and fed a NUL-terminated string held in a MemoryPointer (see
94
+ # with_voice).
95
+ class Voice < FFI::Struct
96
+ layout :name, :pointer, :languages, :pointer, :identifier, :pointer,
97
+ :gender, :uint8, :age, :uint8, :variant, :uint8, :xx1, :uint8,
98
+ :score, :int, :spare, :pointer
99
+ end
100
+
101
+ # phonememode: always IPA; sentinel separator only in sep mode. Used by
102
+ # the phonemize pipeline (stitch substitutes the sentinel afterwards).
103
+ module_function
104
+
105
+ def phonememode(phone_sep)
106
+ mode = PHONEME_IPA
107
+ mode |= (SEP_CHAR.ord << 8) unless phone_sep.empty?
108
+ mode
109
+ end
110
+
111
+ # espeak_Initialize must run once before anything else. Idempotent.
112
+ # AUDIO_OUTPUT_SYNCHRONOUS: no audio device is touched; DONT_EXIT turns a
113
+ # missing espeak-ng-data into a return error instead of a brutal exit().
114
+ def init!
115
+ return true if @initialized
116
+ rate = espeak_Initialize(AUDIO_OUTPUT_SYNCHRONOUS, 0, nil,
117
+ INITIALIZE_DONT_EXIT)
118
+ if rate == EE_INTERNAL_ERROR
119
+ raise "espeak_Initialize failed: is espeak-ng-data installed?"
120
+ end
121
+ @initialized = true
122
+ end
123
+ end
124
+
125
+ # ---------------------------------------------------------------- public API
126
+
127
+ # Input: a String (one utterance) or an Array of words.
128
+ # Returns a String (or an Array of Strings, matching the input shape).
129
+ #
130
+ # Options:
131
+ # language: espeak voice, default "en-us"
132
+ # phone_sep: inserted between phones, default "" (mɑ̃ʒe)
133
+ # word_sep: joins multi-word utterances, default " "
134
+ # strip_stress: remove ˈ ˌ from the output, default true
135
+ # language_switch: :remove (default) or :keep
136
+ # workers: accepted, ignored (kept for API compatibility; the
137
+ # library is called in-process and is not thread-safe)
138
+ def phonemize(words, language: DEFAULT_LANGUAGE, phone_sep: "",
139
+ word_sep: " ", strip_stress: true, language_switch: :remove,
140
+ workers: nil)
141
+ if words.is_a?(Array)
142
+ phonemize_batch(words, language:, phone_sep:, word_sep:,
143
+ strip_stress:, language_switch:)
144
+ else
145
+ phonemize_batch([words], language:, phone_sep:, word_sep:,
146
+ strip_stress:, language_switch:).first
147
+ end
148
+ end
149
+
150
+ def clear_cache
151
+ CACHE.clear
152
+ end
153
+
154
+ # CLI-wrapper compatibility: raw transcription of one utterance. Returns
155
+ # the raw IPA (stress marks, liaison hyphens and language tags intact).
156
+ #
157
+ # Mirrors espeak_raw of the phonemizer gem, where --sep=<char> makes espeak
158
+ # itself separate the phones with the user's character (word boundaries come
159
+ # out as double spaces in sep mode). Single-byte separators are passed
160
+ # through as-is via bits 8-23 of phonememode — the same setting the CLI's
161
+ # --sep uses — verified to produce byte-identical output for sep=" ".
162
+ # Multi-byte separators are inexpressible at the espeak level (the field is
163
+ # one byte; --sep has the same limit), so they fall back to the sentinel and
164
+ # are then substituted; the raw output then differs from the CLI only in
165
+ # word boundaries being spaced once.
166
+ #
167
+ # The CLI's trailing newline (espeak prints a whole line) is not reproduced;
168
+ # all consumers of this value, in both implementations, strip surrounding
169
+ # whitespace before use.
170
+ def espeak_raw(text, language, phone_sep: nil)
171
+ with_voice(language) do
172
+ text = text.to_s
173
+ sep = phone_sep.to_s
174
+
175
+ out =
176
+ if sep.empty?
177
+ transcribe_piece(text, LibEspeak::PHONEME_IPA)
178
+ elsif sep.bytesize == 1
179
+ # CLI parity: pass the separator byte through in bits 8-23 of
180
+ # phonememode, exactly what --sep=<char> does at the espeak level.
181
+ mode = LibEspeak::PHONEME_IPA | (sep.getbyte(0) << 8)
182
+ transcribe_piece(text, mode)
183
+ else
184
+ # Multi-byte separator: impossible at the espeak level.
185
+ # Transcribe with the sentinel, then substitute the user's
186
+ # separator.
187
+ out = transcribe_piece(text, LibEspeak.phonememode(sep))
188
+ out.gsub(LibEspeak::SEP_CHAR, sep)
189
+ end
190
+
191
+ warn "WARNING: espeak-ng returned no output for: #{text[0, 40].inspect}" \
192
+ if out.strip.empty? && !text.strip.empty?
193
+
194
+ out
195
+ end
196
+ end
197
+
198
+ # ---------------------------------------------------------------- pipeline
199
+
200
+ def phonemize_batch(words, language:, phone_sep:, word_sep:,
201
+ strip_stress:, language_switch:)
202
+ # Guard for nil: only copy when some element isn't already a String
203
+ words = words.map(&:to_s) unless words.all? { |w| w.is_a?(String) }
204
+
205
+ # Cache: word forms repeat constantly across a lexicon. Options are fixed
206
+ # for the whole call, so the key is a pure function of the word — memoize
207
+ # it to build each key array only once per unique word.
208
+ key_of = Hash.new do |h, w|
209
+ h[w] = [w, language, phone_sep, word_sep, strip_stress, language_switch]
210
+ end
211
+ todo = words.uniq.reject { |w| CACHE.key?(key_of[w]) }
212
+
213
+ unless todo.empty?
214
+ with_voice(language) do
215
+ # Punctuation: split utterances on punctuation, transcribe only the
216
+ # clean chunks, then splice the punctuation back by position
217
+ chunks = todo.map { |w| split_punctuation(w) }
218
+
219
+ mode = LibEspeak.phonememode(phone_sep)
220
+ clean_words = chunks.flatten.reject { |p| PUNCT.match?(p) }.uniq
221
+ ipa_map = clean_words.each_with_object({}) do |piece, map|
222
+ map[piece] = transcribe_piece(piece, mode)
223
+ end
224
+
225
+ todo.each_with_index do |w, i|
226
+ CACHE[key_of[w]] = stitch(chunks[i], ipa_map,
227
+ phone_sep:, word_sep:, strip_stress:,
228
+ language_switch:)
229
+ end
230
+ end
231
+ end
232
+
233
+ words.map { |w| CACHE[key_of[w]] }
234
+ end
235
+
236
+ # utterance -> [punct, word, punct, word, ...]
237
+ def split_punctuation(word)
238
+ word.split(PUNCT).reject(&:empty?)
239
+ end
240
+
241
+ # Reassemble one utterance: punctuation back in place, separators applied.
242
+ # ipa_map maps each chunk piece to its raw IPA transcription (looked up by
243
+ # content).
244
+ def stitch(parts, ipa_map, phone_sep:, word_sep:, strip_stress:,
245
+ language_switch:)
246
+ parts.map do |part|
247
+ next part if PUNCT.match?(part) # punctuation: verbatim
248
+ t = ipa_map[part] || ""
249
+ t = t.gsub(LANG_SWITCH, "") if language_switch == :remove
250
+ # Always strip stress marks; strip the liaison hyphen only in
251
+ # contiguous mode, where espeak emits it (e.g. "les amis" -> "le-z
252
+ # ami"). In sep mode we transcribed with the sentinel, so a "-" in the
253
+ # user's phone_sep must survive.
254
+ if strip_stress
255
+ t = t.delete("ˈˌ")
256
+ t = t.delete("-") if phone_sep.empty?
257
+ end
258
+ if phone_sep.empty?
259
+ # contiguous phones: a plain space is a word boundary
260
+ t = t.gsub(" ", word_sep)
261
+ else
262
+ # Sep mode: the \x01 sentinel separates phones, a plain space marks a
263
+ # word boundary. Substituted in a single pass, so phone_sep and
264
+ # word_sep may contain ANY characters — including spaces — without
265
+ # collision.
266
+ t = t.gsub(/[\x01 ]/) do |ch|
267
+ ch == LibEspeak::SEP_CHAR ? phone_sep : word_sep
268
+ end
269
+ end
270
+ t
271
+ end.join
272
+ end
273
+
274
+ # ---------------------------------------------------------------- internals
275
+
276
+ # Voice selection in libespeak-ng is global state; set it before
277
+ # transcribing. Initialization guaranteed: every entry point goes through
278
+ # here.
279
+ #
280
+ # Mirrors the espeak-ng CLI: SetVoiceByName recognizes installed voices and
281
+ # variants; pure language codes it doesn't know ("fr-fr" on a system that
282
+ # only installs "fr") fall back to SetVoiceByProperties, which matches on
283
+ # the language (plus dialect) only.
284
+ #
285
+ # The languages field must be a NUL-terminated char*: Ruby's :string
286
+ # coercion only applies to function ARGUMENTS, not to struct fields, and
287
+ # :string struct fields are read-only in FFI — hence the explicit
288
+ # MemoryPointer here. `lang` is kept alive on the local stack for the whole
289
+ # call, which satisfies the struct's lifetime requirement.
290
+ def with_voice(language)
291
+ LibEspeak.init!
292
+ rc = LibEspeak.espeak_SetVoiceByName(language)
293
+ if rc != LibEspeak::EE_OK
294
+ lang = FFI::MemoryPointer.from_string(language) # NUL-terminated
295
+ spec = LibEspeak::Voice.new
296
+ spec[:languages] = lang
297
+ rc = LibEspeak.espeak_SetVoiceByProperties(spec)
298
+ end
299
+ unless rc == LibEspeak::EE_OK
300
+ raise "espeak-ng: cannot select voice #{language.inspect}"
301
+ end
302
+
303
+ yield
304
+ end
305
+
306
+ # Transcribes one piece with the given phonememode. A piece normally holds a
307
+ # single clause (punctuation was split out upstream), but we keep draining
308
+ # clauses as a precaution: a stray terminator would otherwise be silently
309
+ # dropped, and joining the fragments keeps the result sane.
310
+ #
311
+ # Clause fragments are joined with a plain space, like the CLI's one-line
312
+ # IPA output ("words separated by spaces on one line, in input order" — the
313
+ # C API just delivers one clause at a time).
314
+ def transcribe_piece(piece, mode)
315
+ buf = FFI::MemoryPointer.from_string(piece)
316
+ tp = FFI::MemoryPointer.new(:pointer)
317
+ tp.write_pointer(buf)
318
+
319
+ clauses = []
320
+ while (clause = LibEspeak.espeak_TextToPhonemes(
321
+ tp, LibEspeak::CHARS_UTF8, mode
322
+ ))
323
+ clauses << clause
324
+ break if tp.read_pointer.null?
325
+ end
326
+ # FFI returns ASCII-8BIT strings; the IPA payload is UTF-8.
327
+ clauses.join(" ").strip.force_encoding(Encoding::UTF_8)
328
+ end
329
+
330
+ # ---------------------------------------------------------------- CLI
331
+
332
+ # Command-line entry point, mirroring the phonemize bin of the phonemizer
333
+ # gem so this single file serves as both library and command
334
+ # (bin/phonemize_ffi is a thin wrapper around this method). Differences from
335
+ # the gem's bin/phonemize, all imposed by the backend:
336
+ # * no --workers/-j flag (single-threaded FFI backend)
337
+ # * stdin mode added: the CLI has no batch interface on stdin, and the
338
+ # library has no worker pool, so lines from stdin are the natural
339
+ # high-throughput path for this backend.
340
+ def main(argv = ARGV)
341
+ require "optparse"
342
+
343
+ options = {
344
+ language: Phonemizer::DEFAULT_LANGUAGE,
345
+ phone_sep: "",
346
+ word_sep: " ",
347
+ strip_stress: true,
348
+ language_switch: :remove
349
+ }
350
+
351
+ parser = OptionParser.new do |o|
352
+ o.banner = "Usage: #{File.basename($PROGRAM_NAME)} [options] [text | file]"
353
+ o.on("-l", "--language LANG",
354
+ "espeak voice (default: #{Phonemizer::DEFAULT_LANGUAGE})") { |v| options[:language] = v }
355
+ o.on("-p", "--phone-sep SEP",
356
+ "separator between phones (default: none)") { |v| options[:phone_sep] = v }
357
+ o.on("-w", "--word-sep SEP",
358
+ "separator between words in one utterance (default: space)") { |v| options[:word_sep] = v }
359
+ o.on("-s", "--[no-]strip-stress",
360
+ "strip stress marks and liaison hyphens (default: yes)") { |v| options[:strip_stress] = v }
361
+ o.on("-k", "--keep-language-switch",
362
+ "keep inline (en)/(fr) language tags (default: remove)") { options[:language_switch] = :keep }
363
+ o.on("-h", "--help", "show this help") do
364
+ puts o
365
+ exit
366
+ end
367
+ end
368
+ parser.parse!(argv)
369
+
370
+ # Batch stdin mode: one line = one utterance, transcribed in a single batch.
371
+ if argv.empty? && !$stdin.tty?
372
+ lines = $stdin.read.lines.map(&:strip).reject(&:empty?)
373
+ exit if lines.empty?
374
+ Phonemizer.phonemize(lines, **options).each { |line| puts line }
375
+ else
376
+ text = argv.join(" ")
377
+ if File.file?(text)
378
+ lines = File.readlines(text, chomp: true).reject(&:empty?)
379
+ Phonemizer.phonemize(lines, **options).each { |line| puts line }
380
+ else
381
+ puts Phonemizer.phonemize(text, **options)
382
+ end
383
+ end
384
+ end
385
+ end
386
+
387
+ # ---------------------------------------------------------------------------
388
+ # Executable mode: only runs when this file is invoked directly
389
+ # (ruby lib/phonemizer_ffi.rb ... or via a link/execbit), so requiring it from
390
+ # elsewhere stays side-effect free.
391
+ Phonemizer.main(ARGV) if $PROGRAM_NAME == __FILE__
metadata ADDED
@@ -0,0 +1,96 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: phonemizer_ffi
3
+ version: !ruby/object:Gem::Version
4
+ version: 0.1.0
5
+ platform: ruby
6
+ authors:
7
+ - anantone
8
+ bindir: bin
9
+ cert_chain: []
10
+ date: 1980-01-02 00:00:00.000000000 Z
11
+ dependencies:
12
+ - !ruby/object:Gem::Dependency
13
+ name: ffi
14
+ requirement: !ruby/object:Gem::Requirement
15
+ requirements:
16
+ -
17
+ - ~>
18
+ - !ruby/object:Gem::Version
19
+ version: "1.15"
20
+ type: :runtime
21
+ prerelease: false
22
+ version_requirements: !ruby/object:Gem::Requirement
23
+ requirements:
24
+ -
25
+ - ~>
26
+ - !ruby/object:Gem::Version
27
+ version: "1.15"
28
+ - !ruby/object:Gem::Dependency
29
+ name: rake
30
+ requirement: !ruby/object:Gem::Requirement
31
+ requirements:
32
+ -
33
+ - ">="
34
+ - !ruby/object:Gem::Version
35
+ version: "0"
36
+ type: :development
37
+ prerelease: false
38
+ version_requirements: !ruby/object:Gem::Requirement
39
+ requirements:
40
+ -
41
+ - ">="
42
+ - !ruby/object:Gem::Version
43
+ version: "0"
44
+ - !ruby/object:Gem::Dependency
45
+ name: rspec
46
+ requirement: !ruby/object:Gem::Requirement
47
+ requirements:
48
+ -
49
+ - ~>
50
+ - !ruby/object:Gem::Version
51
+ version: "3.13"
52
+ type: :development
53
+ prerelease: false
54
+ version_requirements: !ruby/object:Gem::Requirement
55
+ requirements:
56
+ -
57
+ - ~>
58
+ - !ruby/object:Gem::Version
59
+ version: "3.13"
60
+ description: "A drop-in, API-compatible replacement for the phonemizer gem: same Phonemizer.phonemize API and output, but calling libespeak-ng directly in-process via FFI instead of fork+exec-ing the espeak-ng CLI per word — an order of magnitude faster on large batches, with caching."
61
+ email: self@anantone.me
62
+ executables:
63
+ - phonemize_ffi
64
+ extensions: []
65
+ extra_rdoc_files: []
66
+ files:
67
+ - CHANGELOG.md
68
+ - LICENSE
69
+ - README.md
70
+ - bin/phonemize_ffi
71
+ - lib/phonemizer_ffi.rb
72
+ - lib/phonemizer_ffi/version.rb
73
+ homepage: "https://github.com/anantone/phonemizer_ffi"
74
+ licenses:
75
+ - MIT
76
+ metadata: {}
77
+ rdoc_options: []
78
+ require_paths:
79
+ - lib
80
+ required_ruby_version: !ruby/object:Gem::Requirement
81
+ requirements:
82
+ -
83
+ - ">="
84
+ - !ruby/object:Gem::Version
85
+ version: "3.1"
86
+ required_rubygems_version: !ruby/object:Gem::Requirement
87
+ requirements:
88
+ -
89
+ - ">="
90
+ - !ruby/object:Gem::Version
91
+ version: "0"
92
+ requirements: []
93
+ rubygems_version: 4.1.0
94
+ specification_version: 4
95
+ summary: In-process espeak-ng phonemization via FFI (libespeak-ng)
96
+ test_files: []