@cartanova/qgrid-ai-sdk 2.15.5 → 2.17.0

package/README.ko.md

@@ -394,7 +394,7 @@ Grok language model은 user 이미지 part(https URL 또는 inline bytes)를 버
 | Image | `n`(호출당 최대 3), `aspectRatio`, `xai.resolution`(`1k`/`1.5k`/`2k`), `xai.quality`, `xai.aspect_ratio`(`aspectRatio`가 없을 때) | 참조 `images`는 `referenceImages`(edit endpoint), `xai.output_format`은 `outputFormat`이 되며 `png`, `jpeg`, `webp`만 허용 | `size`, `seed`, `mask`, `xai.sync_mode`, `xai.user` |
 | Video | `duration`(정수 1-15), `aspectRatio`, `xai.resolution`(`480p`/`720p`/`1080p`), `xai.mode: "reference-to-video"`(지원하는 유일한 mode, `xai.referenceImageUrls` 필요), `xai.pollIntervalMs`, `xai.pollTimeoutMs` | `resolution` `WxH`는 `@ai-sdk/xai`와 같이 매핑(`1280x720` → `720p`), prompt `image`는 시작 프레임, `xai.referenceImageUrls`(7개가 아닌 최대 5개)는 `referenceImages`, `xai.lastFrame`(qgrid 확장, `grok-imagine-video-1.5` 전용) | `fps`, `seed`, `xai.keyframes`, `xai.storageOptions`, `xai.referenceVoiceIds`, `xai.user`. `xai.mode` `edit-video`/`extend-video`와 `xai.videoUrl`은 예외 |
 | Transcription | `xai.language` | 오디오 bytes는 base64로, `mediaType`은 파라미터 제거(`audio/webm;codecs=opus` → `audio/webm`) | 나머지 모든 `xai` 키(`audioFormat`, `sampleRate`, `format`, `diarize`, `keyterm`, ...) |
-| Speech | `voice`, `language` | `outputFormat`은 `format`(`mp3`, `wav`, `pcm`, `mulaw`, `alaw`, 그 외 값은 `@ai-sdk/xai`처럼 warning과 함께 `mp3`) | `instructions`, `speed`, 모든 `xai` 키(`sampleRate`, `bitRate`, `withTimestamps`, ...) |
+| Speech | `voice`, `language`, `xai.withTimestamps` | `outputFormat`은 `format`(`mp3`, `wav`, `pcm`, `mulaw`, `alaw`, 그 외 값은 `@ai-sdk/xai`처럼 warning과 함께 `mp3`) | `instructions`, `speed`, 그 밖의 모든 `xai` 키(`sampleRate`, `bitRate`, `replace`, ...) |
 
 **요청 크기.** qgrid 서버는 최대 10 MiB JSON body를 받습니다. inline 미디어는 base64로 전송되어 3분의 1만큼
 커지므로, 서버는 inline 오디오, 각 inline 이미지, 한 요청의 inline 이미지 합계(비디오: `image` + `lastFrame` +
@@ -485,8 +485,9 @@ const speech = await generateSpeech({
 speech.audio.uint8Array;
 ```
 
-- `transcribe`는 bytes로 오디오 형식을 감지하고 실패하면 `audio/wav`를 씁니다. 서버는 `audio/*`가 아닌 값을 거부합니다. transcript에는 segment가 없습니다(`segments: []`).
+- `transcribe`는 bytes로 오디오 형식을 감지하고 실패하면 `audio/wav`를 씁니다. 서버는 `audio/*`가 아닌 값을 거부합니다. Grok transcript는 들린 단어마다 segment 하나를 담습니다(`segments: [{ text, startSecond, endSecond }]`, 2.17.0+). OpenAI realtime transcript에는 없습니다.
 - 음성 합성 텍스트는 최대 15,000자입니다. `voice`, `language`, `outputFormat`을 지정하지 않으면 서버 기본값 `ara`, `en`, `mp3`가 쓰이며, live 검증된 조합은 이것뿐입니다. 다른 voice, language(`@ai-sdk/xai`의 `auto` 포함), codec은 그대로 전달되지만 미검증입니다. 서버가 돌려준 오디오 MIME 타입은 `providerMetadata.qgrid.mediaType`에 있고, AI SDK의 `speech.audio.mediaType`은 bytes에서 추정한 값입니다.
+- 단어 타이밍: `providerOptions: { xai: { withTimestamps: true } }`(2.16.0+, Grok) 또는 `providerOptions: { qgrid: { withTimestamps: true } }`(2.17.0+, Grok·OpenAI)를 주면 글자별 타이밍이 `providerMetadata.qgrid.audioTimestamps`에 담깁니다. `graphChars`는 입력 글자(공백·문장부호·음성 태그 포함), `graphTimes`는 같은 인덱스의 `[start, end]` 초이고, 전체 길이는 `providerMetadata.qgrid.duration`입니다. `@ai-sdk/xai`가 `providerMetadata.xai`에 담는 모양과 같습니다. `providerMetadata.qgrid.timestampsSource`는 Grok이면 `"provider"`(Grok이 직접 잰 값), OpenAI면 `"aligned"`입니다. OpenAI는 서버가 생성된 오디오를 Grok STT로 받아쓰고 단어 단위로 원문과 짝지어 만듭니다(못 들은 단어는 앞뒤 단어 사이 구간을 나눠 가짐). 타이밍 단계가 실패해도 오디오는 돌려주고, 메타데이터에 `timestampsError`와 AI SDK warning을 남깁니다. 텍스트 정규화로 풀린 토큰은 첫 글자가 전체 구간을 가집니다.
 
 ### OpenAI 음성 인식과 합성 (ChatGPT 구독)
 

package/README.md

@@ -404,7 +404,7 @@ Grok language models keep user image parts (https URLs or inline bytes) instead
 | Image | `n` (up to 3 per call), `aspectRatio`, `xai.resolution` (`1k`/`1.5k`/`2k`), `xai.quality`, `xai.aspect_ratio` (used when `aspectRatio` is unset) | reference `images` become `referenceImages` (edit endpoint); `xai.output_format` becomes `outputFormat` and must be `png`, `jpeg`, or `webp` | `size`, `seed`, `mask`, `xai.sync_mode`, `xai.user` |
 | Video | `duration` (integer 1-15), `aspectRatio`, `xai.resolution` (`480p`/`720p`/`1080p`), `xai.mode: "reference-to-video"` (the only supported mode; needs `xai.referenceImageUrls`), `xai.pollIntervalMs`, `xai.pollTimeoutMs` | `resolution` `WxH` is mapped as in `@ai-sdk/xai` (`1280x720` to `720p`); prompt `image` becomes the start frame; `xai.referenceImageUrls` (up to 5, not 7) become `referenceImages`; `xai.lastFrame` (qgrid extension, `grok-imagine-video-1.5` only) | `fps`, `seed`, `xai.keyframes`, `xai.storageOptions`, `xai.referenceVoiceIds`, `xai.user`. `xai.mode` `edit-video`/`extend-video` and `xai.videoUrl` throw |
 | Transcription | `xai.language` | audio bytes become base64 and `mediaType` loses its parameters (`audio/webm;codecs=opus` to `audio/webm`) | every other `xai` key (`audioFormat`, `sampleRate`, `format`, `diarize`, `keyterm`, ...) |
-| Speech | `voice`, `language` | `outputFormat` becomes `format` (`mp3`, `wav`, `pcm`, `mulaw`, `alaw`; anything else falls back to `mp3` with a warning, as in `@ai-sdk/xai`) | `instructions`, `speed`, every `xai` key (`sampleRate`, `bitRate`, `withTimestamps`, ...) |
+| Speech | `voice`, `language`, `xai.withTimestamps` | `outputFormat` becomes `format` (`mp3`, `wav`, `pcm`, `mulaw`, `alaw`; anything else falls back to `mp3` with a warning, as in `@ai-sdk/xai`) | `instructions`, `speed`, every other `xai` key (`sampleRate`, `bitRate`, `replace`, ...) |
 
 **Request size.** The qgrid server accepts JSON bodies up to 10 MiB. Inline media travels as
 base64, which grows by a third, so the server caps inline audio, each inline image, and all
@@ -497,8 +497,9 @@ const speech = await generateSpeech({
 speech.audio.uint8Array;
 ```
 
-- `transcribe` detects the audio type from its bytes and falls back to `audio/wav`; the server rejects anything that is not `audio/*`. Transcripts carry no segments (`segments: []`).
+- `transcribe` detects the audio type from its bytes and falls back to `audio/wav`; the server rejects anything that is not `audio/*`. Grok transcripts carry one segment per heard word (`segments: [{ text, startSecond, endSecond }]`, 2.17.0+); OpenAI realtime transcripts carry none.
 - Speech text is limited to 15,000 characters. Unset `voice`, `language` and `outputFormat` use the server defaults `ara`, `en` and `mp3`, the only combination verified live. Other voices, languages (including `@ai-sdk/xai`'s `auto`) and codecs are passed through but unverified. The server's audio MIME type is in `providerMetadata.qgrid.mediaType`; AI SDK's `speech.audio.mediaType` is sniffed from the bytes.
+- Word timing: `providerOptions: { xai: { withTimestamps: true } }` (2.16.0+, Grok) or `providerOptions: { qgrid: { withTimestamps: true } }` (2.17.0+, Grok or OpenAI) returns per-character timing in `providerMetadata.qgrid.audioTimestamps`: `graphChars` (input characters, spaces, punctuation and speech tags included) and the index-aligned `graphTimes` `[start, end]` seconds, plus `providerMetadata.qgrid.duration`. The shape matches what `@ai-sdk/xai` puts under `providerMetadata.xai`. `providerMetadata.qgrid.timestampsSource` is `"provider"` for Grok, which times its own audio, and `"aligned"` for OpenAI, whose audio the server transcribes with Grok STT and aligns word by word to the text (unheard words share the gap between neighbours). If the timing step fails the audio is still returned, with `timestampsError` in the metadata and an AI SDK warning. Text normalization gives an expanded token's whole span to its first character.
 
 ### OpenAI transcription and speech (ChatGPT subscription)
 

package/dist/index.d.ts

@@ -282,7 +282,9 @@ type QgridOpenAISpeechProviderOptions = {
 /**
  * Grok text-to-speech through `POST /api/qgrid/speak`. `voice`, `language` and `outputFormat`
  * map to the server's `voice`, `language` and `format`; when omitted the server applies
- * `ara` / `en` / `mp3` (the only combination verified live). The model id only labels the log.
+ * `ara` / `en` / `mp3`. The model id only labels the log. `providerOptions.xai.withTimestamps`
+ * returns per-character timing in `providerMetadata.qgrid.audioTimestamps` (`graphChars`,
+ * `graphTimes`), the shape `@ai-sdk/xai` puts under `providerMetadata.xai`.
  */
 declare function createQgridSpeech(modelId: QgridSpeechModel, config?: QgridMediaProviderConfig): SpeechModelV3;
 //#endregion

package/dist/index.js

@@ -1478,7 +1478,7 @@ function isSpeechFormat(value) {
 }
 /**
 * `providerOptions.xai` for Grok speech, field-for-field from `@ai-sdk/xai`
-* `xaiSpeechModelOptionsSchema`. The qgrid server takes none of them; every set key warns.
+* `xaiSpeechModelOptionsSchema`. Only `withTimestamps` is honored; every other set key warns.
 */
 const xaiSpeechOptionsSchema = z.object({
 	sampleRate: z.union([
@@ -1520,12 +1520,42 @@ const OPENAI_SPEECH_FORMATS = ["wav", "pcm"];
 function isOpenAISpeechFormat(value) {
 	return OPENAI_SPEECH_FORMATS.some((format) => format === value);
 }
+/** `providerOptions.xai` keys honored by `qgrid.speech("grok/...")`. */
+const XAI_SPEECH_HONORED = /* @__PURE__ */ new Set(["withTimestamps"]);
 const speakOutputSchema = z.object({
 	audio: z.string(),
 	mimeType: z.string(),
 	tokenName: z.string(),
-	requestLogId: z.number().optional()
+	requestLogId: z.number().optional(),
+	audioTimestamps: z.object({
+		graphChars: z.array(z.string()),
+		graphTimes: z.array(z.tuple([z.number(), z.number()]))
+	}).optional(),
+	durationSec: z.number().optional(),
+	timestampsSource: z.enum(["provider", "aligned"]).optional(),
+	timestampsError: z.string().optional()
 });
+/**
+* `providerOptions.qgrid.withTimestamps` works for both providers (OpenAI has no official option
+* for it). `parse*QgridOptions` drop unknown keys, so it is read here directly.
+*/
+const qgridTimestampsOptionSchema = z.looseObject({ withTimestamps: z.boolean().optional() });
+function qgridWantsTimestamps(raw) {
+	return qgridTimestampsOptionSchema.safeParse(raw).data?.withTimestamps === true;
+}
+/** Shared speech metadata: timing fields plus a warning when requested timing could not be made. */
+function speechTimingMetadata(data, metadata, warnings) {
+	if (data.audioTimestamps) metadata.audioTimestamps = data.audioTimestamps;
+	if (data.durationSec !== void 0) metadata.duration = data.durationSec;
+	if (data.timestampsSource) metadata.timestampsSource = data.timestampsSource;
+	if (data.timestampsError) {
+		metadata.timestampsError = data.timestampsError;
+		warnings.push({
+			type: "other",
+			message: `timestamps unavailable: ${data.timestampsError}`
+		});
+	}
+}
 function base64ToBytes(data) {
 	if (typeof Buffer !== "undefined") return new Uint8Array(Buffer.from(data, "base64"));
 	const binary = atob(data);
@@ -1536,7 +1566,9 @@ function base64ToBytes(data) {
 /**
 * Grok text-to-speech through `POST /api/qgrid/speak`. `voice`, `language` and `outputFormat`
 * map to the server's `voice`, `language` and `format`; when omitted the server applies
-* `ara` / `en` / `mp3` (the only combination verified live). The model id only labels the log.
+* `ara` / `en` / `mp3`. The model id only labels the log. `providerOptions.xai.withTimestamps`
+* returns per-character timing in `providerMetadata.qgrid.audioTimestamps` (`graphChars`,
+* `graphTimes`), the shape `@ai-sdk/xai` puts under `providerMetadata.xai`.
 */
 function createQgridSpeech(modelId, config) {
 	const provider = voiceProviderOf(modelId, LABEL$2);
@@ -1550,7 +1582,7 @@ function createQgridSpeech(modelId, config) {
 			if (!options.text.trim()) throw new Error(`${LABEL$2}: text must not be empty`);
 			if (options.text.length > MAX_TEXT_CHARS) throw new Error(`${LABEL$2}: text is limited to ${MAX_TEXT_CHARS} characters`);
 			const qgridOptions = parseGrokQgridOptions(options.providerOptions?.qgrid, LABEL$2);
-			const xai = parseXaiMediaOptions(options.providerOptions?.xai, xaiSpeechOptionsSchema, /* @__PURE__ */ new Set(), LABEL$2);
+			const xai = parseXaiMediaOptions(options.providerOptions?.xai, xaiSpeechOptionsSchema, XAI_SPEECH_HONORED, LABEL$2);
 			const warnings = [];
 			let format;
 			if (options.outputFormat !== void 0) {
@@ -1580,6 +1612,7 @@ function createQgridSpeech(modelId, config) {
 			if (options.voice !== void 0) requestParams.voice = options.voice;
 			if (options.language !== void 0) requestParams.language = options.language;
 			if (format !== void 0) requestParams.format = format;
+			if (xai.options?.withTimestamps === true || qgridWantsTimestamps(options.providerOptions?.qgrid)) requestParams.withTimestamps = true;
 			if (projectName) requestParams.projectName = projectName;
 			const params = {
 				...requestParams,
@@ -1601,6 +1634,7 @@ function createQgridSpeech(modelId, config) {
 				tokenName: data.tokenName
 			};
 			if (data.requestLogId !== void 0) qgridMetadata.requestLogId = data.requestLogId;
+			speechTimingMetadata(data, qgridMetadata, warnings);
 			return {
 				audio: base64ToBytes(data.audio),
 				warnings,
@@ -1621,6 +1655,8 @@ function createQgridSpeech(modelId, config) {
 * (default `marin`, Codex v2 list), `outputFormat` `wav` (default) or `pcm` — other formats
 * warn and fall back to wav, as the official provider falls back to mp3 — `openai.instructions`
 * as style guidance, `openai.speed` and `language` unsupported with the official warning text.
+* `providerOptions.qgrid.withTimestamps` returns timing the server infers by aligning a Grok STT
+* pass over the generated audio (`providerMetadata.qgrid.timestampsSource: "aligned"`).
 */
 function createOpenAISpeech(modelId, serverUrl, projectName) {
 	return {
@@ -1663,6 +1699,7 @@ function createOpenAISpeech(modelId, serverUrl, projectName) {
 			if (format !== void 0) requestParams.format = format;
 			const instructions = options.instructions ?? openai.options?.instructions ?? void 0;
 			if (instructions !== void 0) requestParams.instructions = instructions;
+			if (qgridWantsTimestamps(options.providerOptions?.qgrid)) requestParams.withTimestamps = true;
 			if (projectName) requestParams.projectName = projectName;
 			const params = {
 				...requestParams,
@@ -1684,6 +1721,7 @@ function createOpenAISpeech(modelId, serverUrl, projectName) {
 				tokenName: data.tokenName
 			};
 			if (data.requestLogId !== void 0) qgridMetadata.requestLogId = data.requestLogId;
+			speechTimingMetadata(data, qgridMetadata, warnings);
 			return {
 				audio: base64ToBytes(data.audio),
 				warnings,
@@ -1772,7 +1810,12 @@ const transcribeOutputSchema = z.object({
 	durationSec: z.number().optional(),
 	model: z.string(),
 	tokenName: z.string(),
-	requestLogId: z.number().optional()
+	requestLogId: z.number().optional(),
+	words: z.array(z.object({
+		text: z.string(),
+		start: z.number(),
+		end: z.number()
+	})).optional()
 });
 function normalizeMediaType(mediaType) {
 	const bare = mediaType.split(";")[0].trim().toLowerCase();
@@ -1828,7 +1871,11 @@ function createQgridTranscription(modelId, config) {
 			if (data.requestLogId !== void 0) qgridMetadata.requestLogId = data.requestLogId;
 			return {
 				text: data.text,
-				segments: [],
+				segments: (data.words ?? []).map((word) => ({
+					text: word.text,
+					startSecond: word.start,
+					endSecond: word.end
+				})),
 				language: data.language,
 				durationInSeconds: data.durationSec,
 				warnings,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@cartanova/qgrid-ai-sdk",
-  "version": "2.15.5",
+  "version": "2.17.0",
   "description": "AI SDK LanguageModelV3 and ImageModelV3 provider for qgrid",
   "repository": {
     "type": "git",