@alia.onl/sdk 13.1.0 → 15.2.1

package/CHANGELOG.md

@@ -1,5 +1,195 @@
 # @alia.onl/sdk
 
+## 15.2.1
+
+- **The shipped source compiles under `noUnusedLocals`.** The package ships its `.tsx` source, so a
+  consumer's own `tsc` (an app with `noUnusedLocals`/`noUnusedParameters`, such as the Oxy website)
+  type-checks it and failed on eight unused imports in `AliaChatMessageList` (`Pressable` and the
+  reanimated worklet helpers, left behind by the work-log move). They are gone, and the SDK's
+  tsconfig now turns the two flags on so it cannot regress.
+
+## 15.2.0
+
+- **`AliaChatSheetRef.present(turn?)`.** Given a turn (`{ text, images? }`), the conversation opens
+  over the sheet with that message already sent: the "ask Alia about this" button of a profile, a
+  post or a file (the way X opens a conversation with Grok about a post). Without a turn nothing
+  changes. `AliaTurn` is exported.
+
+## 15.0.0
+
+A voice call is a realtime speech-to-speech session through Alia and Kaana, and the composer's
+microphone is transcribed by a Kaana model (Alia ADR 0017). Nothing is recognized by the browser
+or the operating system any more, anywhere in the SDK.
+
+- **Breaking: `useVoiceRoom` is a call over Alia's `GET /v1/voice/session` WebSocket.** It sends
+  the microphone as PCM16 24 kHz frames and plays the answer's PCM16 frames; turn detection and
+  barge-in are the model's, on the server, and tools, memory and the stored transcript are Alia's.
+  New options: `conversationId` (where the transcript is stored; omitted, each call starts a
+  conversation of its own), `agentId`, `voicePreference`, `lang`, `onMessagesSaved` (refresh the
+  thread), `onApprovalRequested`, and the test seams `createAudioEngine` / `openSocket`. Gone:
+  `sendTurn`, `chatModel`, `model`, `endOfUtteranceMs`, `bargeIn`, `synthesisError`, `turnError`,
+  `turnErrorCode`, `warning`, `warningCode`.
+- **Honest states.** `callState` (`VoiceCallState`) is `connecting` → `listening` (only once the
+  session is ready upstream, the microphone delivered a buffer AND Alia acknowledged the first
+  appended audio) → `user-speaking` → `thinking` → `using-tool` → `speaking` (only when the first
+  answer frame is PLAYING) → … → `ended { reason }` or `error { code }`, plus `reconnecting`
+  while a dropped socket resumes within the session's resume window or Alia re-opens the session
+  upstream. `roomState` (now with `reconnecting`) and `agentState` are its coarse projections;
+  `endedReason` says why a call ended — `call-moved-elsewhere` (the person opened a call
+  elsewhere) is a reason, not an error. `endingSoonMs`, `model` (the serving model's name; never
+  the operator) are new.
+- **Barge-in:** when the server hears the person start speaking, the playback queue is flushed and
+  Alia is told how many milliseconds of the answer were actually heard, so the model remembers
+  what was said, not what it generated.
+- **The audio engine** (`createVoiceAudioEngine`, `VoiceAudioEngine`): web is `getUserMedia` with
+  echo cancellation, noise suppression and gain control plus two `AudioWorklet`s (capture to
+  24 kHz PCM16, jitter-buffered playback) in one `AudioContext` — Firefox gets calls. iOS and
+  Android use **`react-native-audio-api`** (new optional peer, `^0.13.6`), capture and playback in
+  one native stack; iOS runs `playAndRecord` in mode `voiceChat` (voice processing). **A host app
+  needs a new native build** with the library's config plugin. On Android the library exposes no
+  `voiceCommunication` input preset yet, so platform echo cancellation is a gate to measure on
+  device (ADR 0017 §4).
+- **Breaking: `useSpeechToText` and `useSTTStore` are replaced by `useDictation` and
+  `useDictationStore`.** Dictation records a clip (`MediaRecorder` on web, Opus/WebM or MP4 on
+  Safari; the `expo-audio` recorder on iOS/Android, m4a sent as `audio/mp4`) and transcribes it with
+  `POST /v1/audio/transcriptions` when the person stops. States: `idle`, `requesting-permission`,
+  `recording` (only once the recorder started; levels are the recorder's), `transcribing`,
+  `error`. The text arrives through `onText`, never partially; a failed transcription keeps the
+  clip for one `retry()`. Clips stop by themselves at five minutes, and `remainingMs` counts the
+  last 30 s. `requestTranscription` and `DictationError` are exported. Dictation works in Firefox
+  now.
+- **`AliaComposer`** dictates with `useDictation`: the transcription is appended to the draft the
+  person has when it arrives, a failure is said with a Retry that re-sends the same clip, and the
+  time left shows over the composer in the last 30 s. New prop `apiUrl`; new label
+  `dictationRemaining`.
+- **Breaking: `VoiceErrorCode` is rewritten.** The recognizer's codes (`speech-*`,
+  `microphone-*`, `recognizer-*`, `no-audio-input`, `turn-failed`, `no-answer`, `not-played`,
+  `synthesis-*`) are gone. It is now Alia's call codes (`not-authenticated`,
+  `credential-not-supported`, `plan-required`, `credits-exhausted`, `agent-unavailable`,
+  `invalid-request`, `resume-unavailable`, `realtime-unavailable`, `realtime-limited`,
+  `realtime-rejected`, `realtime-failed`), the device's (`mic-denied`, `mic-unavailable`,
+  `connection-lost`, `voice-unavailable`) and dictation's (`too-long`,
+  `transcription-unavailable`, `transcription-limited`, `transcription-rejected`,
+  `transcription-failed`). An app with its own translations replaces its keys.
+- **Removed:** `createAliaVoiceTurnSender` and the `VoiceTurn*` types, the device recognizers
+  (`speech-recognition*`, `chooseSpeechRecognizer`), `isInterruption` and the rest of
+  `voice-text`, and the **`expo-speech-recognition` peer** — a host app removes it and its config
+  plugin. Read-aloud (`useTTS`, `/v1/audio/speech`) is unchanged.
+- `VoiceSession` takes `conversationId`, `agentId`, `locale` and `voice`; `VoiceControls` takes
+  `callState` and shows `reconnecting` and `usingTool` (new labels).
+- The `./voice` entry still holds the call: `scripts/check-entry-isolation.mjs` now also proves the
+  root entry never reaches `react-native-audio-api` (nor `expo-speech-recognition`).
+
+## 14.1.0
+
+An "Ask Alia" button can show her live avatar and react to what she is doing.
+
+- **`AliaAvatarIcon`**: a Bloom icon component (`icon={AliaAvatarIcon}`) that draws the live avatar
+  at the icon size under its own `AgentAvatarProvider` on `ALIA_CHARACTER_RUNTIME_URL`, not interactive.
+- **`useAliaActivity()`** returns `'idle' | 'thinking' | 'working' | 'writing'`, the busiest of the
+  turns running in this app (`useAliaChat` now publishes its lifecycle to a shared store) and the
+  server's `alia:activity` events for the signed-in person (one shared `socket.io-client` connection
+  per API origin, none while signed out, closed with the last consumer). New dependency:
+  `socket.io-client`. Needs the API that emits `alia:activity` (an older API simply never sends it:
+  buttons then follow local turns only).
+- The header mark and the buttons read a turn from one function (`turnActivityState`).
+
+## 14.0.0
+
+Read-aloud asks the server every time, and the server answers a clip it already made from storage:
+a message read aloud twice, or a sentence voice mode says again, is synthesized and paid for once.
+
+- **Breaking: `useTTS().readAloud(messageId, text, conversationId?)`** loses its fourth argument,
+  `audioUrl`. A message no longer carries a stored clip of its own: `POST /v1/audio/speech` finds
+  the clip by its text, voice, speed and speech model, for the signed-in person, and answers a
+  stored one at once. There is no "play the stored link, remake it if it has gone" path any more.
+- **Breaking: `ChatMessage.audioUrl` is removed.** `GET /conversations/:id` no longer sends it.
+- `requestSpeechClip`'s `conversationId` + `messageId` now file the clip under that conversation,
+  so deleting or clearing the conversation deletes it. The route still answers 404 for a message
+  the person does not hold, and `requestId` is present only when the clip was made by that request.
+
+A voice call says "Listening…" only while something listens.
+
+- **`useVoiceRoom` stays `connecting` until the recognizer has started** (`roomState` is
+  `'connected'` and `agentState` `'listening'` from the first session's start, not from
+  `connect()`), and a session that does not start within 8 s ends the call as
+  `recognizer-not-started`.
+- **An `aborted` nobody asked for is a failure.** Chrome runs one microphone recognition per
+  browser and a new one aborts the live one, in any tab; the engines now report that as the
+  failure code `interrupted` (only the abort the caller asked for stays silent), and a call cut
+  off three times in a row ends as `recognizer-aborted` instead of restarting forever. The web
+  engine serializes its own sessions, so a page never aborts itself.
+- **The call watches itself while it listens:** voice-level sound for 5 s with no word back ends
+  it as `recognizer-unresponsive`; five sessions in a row that end without listening end it with
+  their error's code; a microphone silent for 10 s raises **`warning` / `warningCode`**
+  (`no-audio-input`) without ending the call. Recognizer errors are logged as a structured
+  `console.warn`, never with the transcript.
+- **`VoiceErrorCode` gains** `recognizer-not-started`, `recognizer-unresponsive`,
+  `recognizer-aborted` and `no-audio-input` (an app with its own translations adds them);
+  `VoiceControls` takes a `notice` it shows in place of "Listening…". Dictation reports an
+  interrupted session as `recognizer-aborted` too.
+
+## 13.1.1
+
+- **The avatar flights load the runtime too.** 13.1.0 mounted the character runtime provider inside
+  the chat, but the sheet's avatar-flight layer (the avatar that flies from the welcome into the
+  header) is drawn outside it, so it asked the host for its own `/bloom-character/runtime.mjs` and got
+  the app's HTML back ("Expected a JavaScript-or-Wasm module script but the server responded with
+  a MIME type of text/html"). `AliaChatSheet` and `AliaChatScreen` now wrap everything they draw in the
+  provider, once; `AliaChatContent` no longer mounts one (it is not an entry point on its own).
+
+## 13.2.0
+
+One composer for the sheet, the screen and the Alia app: the app's composer moved here and the
+app consumes it.
+
+- **`AliaComposer`** (exported) is Bloom's `ComposerPanel` plus what every Alia surface needs the
+  same way: files by paste and drop (read with real progress, retried, refused with a sentence),
+  dictation, the suggestions over the field and the questions Alia asked, docked above it. What
+  differs per surface is a prop: `addMenu` / `onAddMenuSelect`, `modes` / `mode` /
+  `onModeChange` (a power-level selector), `attachments` / `onAddAttachment` /
+  `onRemoveAttachment` (a composer given no `onAddAttachment` takes no paste or drop),
+  `attachmentKinds` (`['image']` refuses a document with a sentence), `suggestions`, `questions`,
+  `status`, `emptyAction`, `accessory`, `onKeyPress`, `speechLanguage`, `labels`, `onError`.
+  `AliaChatContent` (so the sheet and the screen) renders it with photos, voice and suggestions;
+  a host gets today's behaviour with no setup.
+- **The sheet's composer is the panel, not the pill.** The pill has no mode selector, status tab
+  or question slot, which the Alia app needs; the sheet and the app now draw the same card. The
+  agent-working light (`ComposerLoader`) is not drawn around it, as in the app: a turn in flight
+  is its stop control and the thread's thinking line.
+- **Photos are `AliaAttachment`s** (exported type): the chat's picked photos wait as the same
+  tiles a dropped or pasted image becomes, with the same remove and error states.
+- **`AliaChatLabels` gains** `composerMessage`, `composerAddMenu`, `composerModes`,
+  `composerVoice`, `composerSend`, `composerStop`, `composerRemove`, `composerRetry`,
+  `composerUntitled`, `fileReadFailed`, `fileTooLarge`, `fileEmpty`, `fileUnsupported`,
+  `dropHint`, `dropUnavailable` and `voiceError(code)`, all with English defaults. A nested
+  `AliaChatLabelsProvider` now refines the labels already in reach instead of resetting them.
+- **`usePendingQuestions(messages, isStreaming)`** (exported) is the "asked above the composer,
+  put away until she asks again" rule both the chat and the app used to keep a copy of.
+- **`classifyIntake` / `MAX_ATTACHMENT_BYTES`** (from `@alia.onl/sdk/tools`) are the rules a file
+  passes to be attached.
+- **The sheet focuses its composer once it has presented.** `AliaChatSheet` hands each sheet's
+  `animatedProgress` (Bloom's own "fully presented" signal, no timer) to its chat, whose composer
+  takes focus when it reads `1`: the person can type at once and the keyboard opens on native. It
+  focuses once per `present()` for the main sheet and for the conversation sheet over it; dragging
+  a sheet part-way and letting go does not take the focus again. `AliaChatContent` takes
+  `presentedProgress` for a host that builds its own sheet; the screen and the app's own screens
+  pass none and keep their focus. Not verified on a device.
+
+A voice call that cannot speak ends and says why, instead of listening on in silence:
+
+- **`requestSpeechClip` throws a `SpeechClipError`** (exported) with the route's HTTP `status`,
+  its typed `code` (`SPEECH_UNAVAILABLE`, `SPEECH_LIMIT_REACHED`, `SPEECH_REJECTED`,
+  `SPEECH_PROVIDER_FAILED`, `SPEECH_GENERATION_FAILED`, `INVALID_REQUEST`, `NOT_AUTHENTICATED`, or
+  `null`), `retryable` and `requestId`. An empty token is `NOT_AUTHENTICATED` without a request.
+- **`useVoiceRoom` ends the call when a clip cannot be made or played**, after the answer has
+  finished streaming into the transcript, with `errorCode` `synthesis-unavailable`,
+  `synthesis-limited`, `synthesis-rejected`, `synthesis-failed`, `not-authenticated` (no session
+  token) or `not-played`; `synthesisError` is the `SpeechClipError` behind it. A turn no longer
+  skips a sentence it could not speak, and `turnErrorCode` is no longer `not-played`.
+- `VoiceErrorCode` gains the four `synthesis-*` codes; an app that translates every code needs
+  their sentences.
+
 ## 13.1.0
 
 The chat works in any app without host setup it did not document in code:

package/README.md

@@ -53,31 +53,39 @@ Two entry points:
 
 ## Speech
 
-Dictation (`useSpeechToText`) and the voice call (`useVoiceRoom`) recognize
-speech **on the device**: the Web Speech API in browsers that have it (Chromium
-and Safari — not Firefox, where `useSpeechToText().isSupported` is `false`), and
-[`expo-speech-recognition`](https://github.com/jamsch/expo-speech-recognition)
-on iOS and Android. No audio is sent to Alia; whether the platform's recognizer
-uses a network service of its own is the platform's (Chrome's does).
-
-On native, add `expo-speech-recognition` (a peer dependency, like `expo-audio`:
-the root entry's mic button reaches it) and its config plugin to your app, then
-make a new native build:
+Nothing is recognized on the device. Speech is recognized by Kaana, through
+Alia (Alia ADR 0017):
+
+- **Dictation** (`useDictation`, the composer's mic) records a clip — the
+  browser's `MediaRecorder`, or the `expo-audio` recorder on iOS and Android —
+  and, when the person stops, sends it to `POST /v1/audio/transcriptions`.
+  The text arrives once, never partially. Five minutes per clip. Free on every
+  Alia plan.
+- **A voice call** (`useVoiceRoom`, `VoiceSession`) is a realtime
+  speech-to-speech session: a WebSocket to Alia's `GET /v1/voice/session`
+  carrying PCM16 24 kHz audio both ways. Turn detection and barge-in are the
+  model's; tools run in Alia; the transcript is stored in the conversation.
+  Voice calls need a paid Alia plan and use credits per minute.
+
+Audio reaches Alia, Kaana and the model's provider; Alia logs no audio and no
+transcript content.
+
+On iOS and Android a call needs
+[`react-native-audio-api`](https://github.com/software-mansion/react-native-audio-api)
+(an optional peer of `@alia.onl/sdk/voice`; the root entry never reaches it)
+and its config plugin, then a new native build:
 
 ```json
-["expo-speech-recognition", {
-  "microphonePermission": "Allow $(PRODUCT_NAME) to hear what you dictate and say in voice calls.",
-  "speechRecognitionPermission": "Allow $(PRODUCT_NAME) to turn your speech into text."
+["react-native-audio-api", {
+  "iosMicrophonePermission": "Allow $(PRODUCT_NAME) to hear what you dictate and say in voice calls.",
+  "iosBackgroundMode": false,
+  "androidForegroundService": false,
+  "androidPermissions": ["android.permission.RECORD_AUDIO"]
 }]
 ```
 
-A voice call is a turn loop, not a realtime session: it listens until the
-person pauses, sends what they said through the ordinary chat path
-(`POST /v1/chat/completions` with `responseMode: "voice"`, or your own
-`sendTurn`), and speaks the answer sentence by sentence through
-`POST /v1/audio/speech` in the product voice (`male` / `female`). Talking over
-the answer interrupts it. Pass `lang` to both hooks to follow your app's
-language.
+Dictation needs only `expo-audio` (already a peer). Pass `lang` to both hooks
+(`speechLanguage` to `AliaComposer`) to follow your app's language.
 
 ## Use
 
@@ -112,6 +120,61 @@ components render. Those events are the product runtime's, not part of the Oxy
 generic inference contract, and this package is written against them on
 purpose: it is a **product** client.
 
+## An "Ask Alia" button with her live avatar
+
+```tsx
+import { AliaAvatarIcon, useAliaActivity } from '@alia.onl/sdk';
+
+<Button icon={AliaAvatarIcon} onPress={openAlia}>Ask Alia</Button>;
+const state = useAliaActivity(); // 'idle' | 'thinking' | 'working' | 'writing'
+```
+
+`AliaAvatarIcon` is a Bloom icon component (`BloomIconComponent`): pass it as `icon` /
+`leadingIcon`. It draws the live 3D avatar at the icon size under its own
+`AgentAvatarProvider` (`ALIA_CHARACTER_RUNTIME_URL`), and plays what Alia is doing.
+
+`useAliaActivity()` is the busiest of two sources, so a button reacts even with the sheet closed:
+
+- a turn running in this app: every `useAliaChat` publishes its lifecycle to a shared store;
+- her runs for the signed-in person elsewhere (background work, another device): the API's
+  `alia:activity` event on the person's `user:<id>` room, over ONE Socket.IO connection per
+  API origin, opened with the Oxy session's token only while someone is signed in and a
+  consumer is mounted, closed with the last one. A run that goes quiet for 90 s is treated as over.
+
+`socket.io-client` is a regular dependency, not a peer: the SDK owns this connection and an
+app should not have to install a transport to use a button.
+
+## The composer
+
+The sheet, the screen and the Alia app share one composer, `AliaComposer`: Bloom's
+`ComposerPanel` with files by paste and drop (read with real progress, retried, refused with a
+sentence), dictation, suggestions over the field and Alia's questions docked above it.
+`AliaChatSheet` and `AliaChatScreen` render it with photos, voice and suggestions, and
+`AliaChatSheet` focuses it once the sheet has presented. To build your own surface, mount it and
+pass what the surface has:
+
+```tsx
+import { AliaComposer } from '@alia.onl/sdk';
+
+<AliaComposer
+  value={draft}
+  onValueChange={setDraft}
+  onSubmit={send}
+  busy={isStreaming}
+  onStop={stop}
+  modes={powerLevels} // the mode selector; omit to hide it
+  mode={level}
+  onModeChange={setLevel}
+  addMenu={groups} // the add menu's rows, and what pressing one does
+  onAddMenuSelect={onAdd}
+  attachments={files} // omit `onAddAttachment` and the composer takes no paste or drop
+  onAddAttachment={addFile}
+  onRemoveAttachment={removeFile}
+  attachmentKinds={['image']} // what your send carries; others are refused with a sentence
+  labels={{ composerSend: 'Enviar' }} // anything omitted is English
+/>;
+```
+
 ## Which endpoint the SDK calls, and why
 
 The hook sends its turns to `POST {apiUrl}/v1/chat/completions`, with `apiUrl`

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@alia.onl/sdk",
-  "version": "13.1.0",
-  "description": "Alia AI chat SDK — reusable chat UI, voice, and streaming components for Oxy apps",
+  "version": "15.2.1",
+  "description": "Alia AI chat SDK — reusable chat UI, voice calls, dictation and streaming components for Oxy apps",
   "main": "src/index.ts",
   "react-native": "src/index.ts",
   "source": "src/index.ts",
@@ -32,6 +32,7 @@
   "dependencies": {
     "clsx": "^2.1.1",
     "react-native-markdown-display": "^7.0.2",
+    "socket.io-client": "^4.8.3",
     "tailwind-merge": "^3.7.0",
     "zustand": "^5.0.15"
   },
@@ -57,7 +58,7 @@
     "nativewind": "^5.0.0-preview.3",
     "expo-clipboard": "^57.0.0",
     "expo-file-system": "^57.0.0",
-    "expo-speech-recognition": "^57.1.0"
+    "react-native-audio-api": "^0.13.6"
   },
   "peerDependenciesMeta": {
     "nativewind": {
@@ -65,6 +66,9 @@
     },
     "expo-clipboard": {
       "optional": true
+    },
+    "react-native-audio-api": {
+      "optional": true
     }
   },
   "devDependencies": {
@@ -72,6 +76,6 @@
     "react-test-renderer": "19.2.3",
     "typescript": "^5.9.3",
     "vitest": "^4.1.10",
-    "expo-speech-recognition": "~57.1.0"
+    "react-native-audio-api": "0.13.6"
   }
 }

package/src/components/AliaAvatarIcon.tsx

@@ -0,0 +1,30 @@
+import { AgentAvatarProvider } from '@oxy.so/bloom/agent-avatar';
+import type { BloomIconComponent } from '@oxy.so/bloom/icons';
+import { useAliaActivity } from '../hooks/useAliaActivity';
+import { ALIA_CHARACTER_RUNTIME_URL } from '../lib/character-runtime';
+import { AliaAvatar } from './AliaAvatar';
+
+const DEFAULT_SIZE = 20;
+
+/**
+ * Alia's live avatar as a Bloom icon: pass it as `icon` / `leadingIcon` to a Bloom `Button` and
+ * the button shows her, reacting as she thinks, works and writes (`useAliaActivity`) — also while
+ * the chat sheet is closed.
+ *
+ * ```tsx
+ * <Button icon={AliaAvatarIcon} onPress={openAlia}>Ask Alia</Button>
+ * ```
+ *
+ * It mounts its own `AgentAvatarProvider` on `ALIA_CHARACTER_RUNTIME_URL`: an avatar outside the
+ * chat's provider would ask the host for `/bloom-character/runtime.mjs` and get its HTML. The
+ * colour (`fill`) is ignored — the likeness is painted — and it is never interactive, so the
+ * button keeps the press. Reduced motion is Bloom's engine's to honour.
+ */
+export const AliaAvatarIcon: BloomIconComponent = ({ width, height }) => {
+  const state = useAliaActivity();
+  return (
+    <AgentAvatarProvider runtimeUrl={ALIA_CHARACTER_RUNTIME_URL}>
+      <AliaAvatar size={width ?? height ?? DEFAULT_SIZE} state={state} />
+    </AgentAvatarProvider>
+  );
+};