@genex-ai/embed-sdk 0.37.0 → 0.38.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -475,12 +475,15 @@ npx genex llm practice
475
475
  ```
476
476
 
477
477
  With that key in the page's URL, `getGenerationModels()`, `generate()`,
478
- `generateStream()`, `judge()`, `requestSpendGrant()` and the budget helpers are
478
+ `generateStream()`, `judge()`, `requestSpendGrant()`, the budget helpers and the
479
+ voice calls (`getVoiceCatalog()`, `transcribe()`, `speak()`) are
479
480
  answered by the helper on your machine, in the same shapes, statuses and error
480
481
  codes the published lane returns, so the code you write against it runs
481
482
  unchanged when published. Answers are made up and free: text starts with
482
483
  `[practice]`, JSON is generated from your `schema`, a stream arrives in pieces,
483
- and a budget is approved by the helper and counts a pretend credit a call.
484
+ a spoken line is a babble about as long as the real one, a recording is heard
485
+ as a stock line, and a budget is approved by the helper and counts a pretend
486
+ credit a call.
484
487
  Flags stage the hard cases: `--decline`, `--limit-credits <n>`,
485
488
  `--fail <code>`, `--latency <ms>`, `--calls-per-minute <n>`, and
486
489
  `--answers <file>` for exact answers (`npx genex llm --help`).
@@ -1042,6 +1045,119 @@ Benchmark on your own credits with `developmentJudge()` from
1042
1045
  billed one whole credit a call — a player's budget, which meters, spends far
1043
1046
  less for the same calls.
1044
1047
 
1048
+ ## Voice
1049
+
1050
+ SDK **0.38.0+**. The game can **hear the player** and **talk back**. `transcribe()` turns a
1051
+ recording into words; `speak()` says a line in one of Genex's voices, streamed
1052
+ while it is made; `createSpeaker()` plays lines back to back, and can speak a
1053
+ model's reply while `generateStream()` is still writing it. Each is a call
1054
+ under a standing budget, billed as used on its meter like `judge()` (never a
1055
+ credit of its own), so voice is **grant-only**: ask for it when you request the
1056
+ budget. A voice budget is paid from the player's Genex credits, and the sheet
1057
+ names every company the player's voice or the game's lines are sent to.
1058
+ **Genex stores no audio**: a recording is passed to the transcription provider
1059
+ and dropped, and a spoken line is streamed and never kept.
1060
+
1061
+ ```ts
1062
+ import { getVoiceCatalog, requestSpendGrant, createVoiceRecorder, transcribe, generateStream, createSpeaker,
1063
+ generationErrorMessage } from '@genex-ai/embed-sdk';
1064
+
1065
+ const catalog = await getVoiceCatalog();
1066
+ if (!catalog.enabled) return useTypedInput(); // no voice here: typing still works
1067
+
1068
+ // From the player's click, as every budget request.
1069
+ const tier = catalog.speech.models[0]!.tier; // tiers and voices come from the catalog
1070
+ const voice = catalog.speech.voices.find((v) => v.tiers.includes(tier))!.id;
1071
+ const budget = await requestSpendGrant({
1072
+ models: [model.id],
1073
+ perCallMaxCredits: 8, perCallEstimateCredits: 2,
1074
+ disclosure: { periodLabel: 'minute', estimatedCallsPerPeriod: 4, estimatedCreditsPerPeriod: 8 },
1075
+ voice: {
1076
+ speech: { tiers: [tier], estimatedCharactersPerPeriod: 1200 }, // the game speaks
1077
+ transcription: { estimatedSecondsPerPeriod: 60 }, // the game listens
1078
+ },
1079
+ });
1080
+ const grantId = budget.grantId!;
1081
+
1082
+ const recorder = createVoiceRecorder({ onLevel: (level) => micMeter.set(level) });
1083
+ const speaker = createSpeaker({ grantId, voice, maxCharacters: catalog.limits.speech.maxCharacters,
1084
+ onLineStart: (line) => subtitle.show(line.text),
1085
+ onLineEnd: (line) => { if (line.error) showNotice(generationErrorMessage(line.error)); } });
1086
+
1087
+ talk.onpointerdown = () => recorder.start().catch((error) => showNotice(generationErrorMessage(error.code)));
1088
+ talk.onpointerup = async () => {
1089
+ const audio = await recorder.stop().catch(() => null); // `not_recording`: the start was refused
1090
+ if (!audio) return;
1091
+ const heard = await transcribe({ grantId, audio });
1092
+ if (heard.status !== 'succeeded') return showNotice(generationErrorMessage(heard.error));
1093
+ if (!heard.text) return; // nobody spoke
1094
+ await generateStream({ grantId, modelId: model.id, prompt: heard.text, outputFormat: 'text', maxCredits: 8,
1095
+ onDelta: (piece) => speaker.feed(piece) }); // each sentence is spoken as it closes
1096
+ await speaker.end(); // the rest, then wait until it has played
1097
+ };
1098
+ ```
1099
+
1100
+ - **Cast from the catalog.** `getVoiceCatalog()` lists the voices (a one-line
1101
+ description to cast from, a gender, and the tiers that can speak each), one
1102
+ speech model per tier (`premium`, `standard`, `economy`: a game names a
1103
+ tier, never a vendor), the transcription models, the limits (characters per
1104
+ line, seconds per recording) and estimated prices. Never type a voice id or a
1105
+ tier from memory. It answers `{ enabled: false }` wherever voice cannot run (a
1106
+ stand without it, local test mode without the practice helper, the app, an
1107
+ untrusted frame, no player) and never throws. Keep typed input as the
1108
+ fallback: a player may refuse the microphone, have none, or play where voice
1109
+ is off.
1110
+ - **Push to talk.** `createVoiceRecorder({ maxSeconds?, onLevel?, deviceId? })`:
1111
+ `start()` on the press asks for the microphone, `stop()` on the release
1112
+ resolves a 16 kHz mono WAV `Blob`, `cancel()` throws it away. It stops itself
1113
+ at `maxSeconds` (default 30, at most 60) and the next `stop()` returns that
1114
+ recording. A failure rejects with `error.code` `microphone_denied`,
1115
+ `microphone_unavailable` or `not_recording`. The microphone and the audio
1116
+ graph are let go on stop, cancel and the cut-off. On its Genex page a game is
1117
+ given the microphone only when its build declares it: add
1118
+ `"genex": { "capabilities": ["microphone"] }` to `package.json`.
1119
+ - **`transcribe({ grantId, audio, language?, model?, idempotencyKey?, signal? })`**
1120
+ takes one 16-bit PCM WAV (anything else is `audio_invalid`, before the
1121
+ network) and resolves `text` (`''` when nobody spoke), `language`,
1122
+ `durationMs` and `billing`. Reuse `idempotencyKey` to retry the same recording
1123
+ without paying twice; `pending` with `wait_timeout` means that call is still
1124
+ running: send it again with the same key to read its answer.
1125
+ - **`speak({ grantId, text, voice, tier?, style?, language?, onStart?, onAudio?, signal? })`**
1126
+ streams 16-bit PCM (24 kHz mono; `onStart` names the format) and resolves the
1127
+ line's receipt with all of its audio (`audio`, `wav()`). A spoken line is
1128
+ never replayed: its key is used once, and a reused one is
1129
+ `idempotency_conflict`. A stream that is cut resolves `unknown` with
1130
+ `provider_outcome_unknown` and the audio that arrived; `signal` stops the
1131
+ reading (`request_aborted`), and the line is still made and charged.
1132
+ - **`createSpeaker({ grantId, voice, tier?, style?, language?, onLineStart?, onLineEnd?, maxCharacters? })`**
1133
+ queues lines (`say(text)`) or a streamed reply (`feed(delta)` then `end()`),
1134
+ one request at a time and in order, each made while the one before it plays,
1135
+ so speech has no gaps. A line past `maxCharacters` is split at sentence ends,
1136
+ then at spaces. A line that fails (`grant_limit_reached`, `voice_busy`, …) is
1137
+ reported through `onLineEnd` and the queue moves on; `stop()` silences at once
1138
+ and drops the queue, and each dropped line ends `canceled` with
1139
+ `game_stopped`. **Browsers play audio only after the player has interacted
1140
+ with the page**: create the speaker, or call `say()`, from a click or a key
1141
+ press.
1142
+
1143
+ The refusals to handle as game states: `voice_requires_grant` (no budget),
1144
+ `voice_not_enabled` (the budget was approved without voice, or without the half
1145
+ this call needs: request a new one), `voice_requires_credits`,
1146
+ `voice_not_found` (a voice that is not in the catalog or not on that tier),
1147
+ `voice_busy` and `voice_paused` (the platform's, retry later), `text_too_long`,
1148
+ `audio_invalid`, `audio_too_long`, `audio_too_short`, and every budget code
1149
+ above. `generationErrorMessage()` has a sentence for each.
1150
+
1151
+ The wire, for other clients: `GET /api/runtime/voice` is the catalog (404 where
1152
+ the stand has no voice). `POST /api/runtime/grants/:id/transcriptions` takes the
1153
+ WAV bytes as its body (`Content-Type: audio/wav`), an `Idempotency-Key` header
1154
+ and optional `language` and `model` query parameters, and answers JSON.
1155
+ `POST /api/runtime/grants/:id/speech` takes
1156
+ `{ idempotencyKey, text, voice, tier?, style?, language? }` and answers
1157
+ `text/event-stream`: one `start` `{ generationId, encoding, sampleRate, channels }`,
1158
+ `audio` `{ data }` events (base64 PCM, whole samples, in order) and one `done`
1159
+ carrying the receipt. Both take the embed token as a bearer.
1160
+
1045
1161
  ## Your subscription and the Genex player watcher
1046
1162
 
1047
1163
  Both approval sheets can offer the player their own Claude or ChatGPT plan