@hoanghong/speaking-core 0.5.0 → 0.6.0

package/CHANGELOG.md

@@ -1,3 +1,9 @@
+## 0.6.0
+
+- Versioned overall local pronunciation score (harmonic mean of accuracy and completeness).
+- Explicit score scale, availability, components, exclusions and calibration status.
+- New clients accept older servers without inventing an overall score.
+
 ## 0.5.0
 
 - Optional multi-criteria local pronunciation diagnostics with methods, limitations and unavailable scores.

package/README.md

@@ -1,113 +1,174 @@
 # @hoanghong/speaking-core
 
-SDK online cho người tự làm UI. Bản test 0.5.0 cài bằng `npm install ./vendor/hoanghong-speaking-core-0.5.0.tgz` từ bộ ZIP, chưa publish registry. Angular dùng @hoanghong/speaking-angular với component có sẵn; Flutter dùng fsel_speaking. MIT.
+TypeScript SDK for custom speaking interfaces: realtime transcription, audio upload and US English pronunciation diagnostics. Models run on the service. The SDK manages connections and short-lived sessions; no service URL or service token is needed. MIT license.
 
-## Đánh giá phát âm tiếng Anh Mỹ
+## Install from npm
+
+```bash
+npm install @hoanghong/speaking-core@0.6.0
+```
+
+## Install the supplied version 0.6.0 archive
+
+Copy the tarball from the release ZIP into your application's vendor directory:
+
+```bash
+npm install ./vendor/hoanghong-speaking-core-0.6.0.tgz
+```
+
+These instructions use the supplied archive and do not assume registry publication. Angular applications may use @hoanghong/speaking-angular; Flutter applications may use fsel_speaking.
+
+## Assess pronunciation from a file
+
+Ask the learner to read the exact reference passage. Pass a selected File or recorded Blob to assessAudio. Transcription alone does not assess pronunciation.
 
 ```ts
-import { PronunciationClient, createSpeakingService } from '@hoanghong/speaking-core';
-const pronunciation = new PronunciationClient(createSpeakingService().pronunciation);
-export async function assess(file: File) {
-  return pronunciation.assessAudio(file, {
-    referenceText: 'cat', expectedPhonemes: ['k', 'æ', 't'], filename: file.name,
-  });
+import {
+  createSpeakingService, PronunciationClient, PronunciationAssessment, SpeakingError,
+} from '@hoanghong/speaking-core';
+
+const config = createSpeakingService();
+// If FSEL supplied an application key, use instead:
+// const config = createSpeakingService({ appKey: 'YOUR_APP_KEY' });
+const pronunciation = new PronunciationClient(config.pronunciation);
+
+export async function assessPronunciation(
+  file: File,
+  referenceText: string,
+  signal?: AbortSignal,
+): Promise<PronunciationAssessment> {
+  try {
+    return await pronunciation.assessAudio(file, {
+      referenceText,
+      filename: file.name,
+      signal,
+    });
+  } catch (error) {
+    if (error instanceof SpeakingError) {
+      console.error(error.code, error.status, error.message);
+    }
+    throw error; // Let the UI display the error and offer a deliberate retry.
+  }
+}
+
+export function describeOverall(result: PronunciationAssessment): string {
+  const value = result.overall;
+  return value?.status === 'estimated' && value.score != null
+    ? `${value.score.toFixed(1)} / ${value.scale} (local estimate)`
+    : value?.reason ?? 'Unavailable: upgrade the server if needed';
+}
+
+export function describeAccuracy(result: PronunciationAssessment): string {
+  const accuracy = result.breakdown?.accuracy;
+  console.log(result.comparisons, result.warnings);
+  return accuracy?.status === 'estimated' && accuracy.score != null
+    ? `${accuracy.score.toFixed(1)} / 100 (estimate)`
+    : 'Unavailable';
 }
 ```
 
-Audio tối đa 120 giây/25 MiB, reference tối đa 4096 ký tự; timeout client 11 phút.
-Đọc cả đoạn: thay referenceText bằng đoạn mẫu; bỏ expectedPhonemes khi server có eSpeak.
-Các âm tùy chọn là token vocabulary, không tách IPA theo ký tự. Mỗi lần gọi cấp phiên riêng,
-không tự retry. AbortSignal được hỗ trợ. Server phải triển khai endpoint/model phát âm;
-bộ cài này chưa xác nhận hosted endpoint đã sẵn sàng. Gateway riêng dùng cấu hình
-`createSpeakingGateway(...).pronunciation`, giữ credential của service ở backend.
-Response typed PronunciationAssessment: âm mẫu/nhận dạng, comparisons, timestamp ước lượng,
-formant/pitch/intensity/PSD và warnings. `calibrated=false`, không dùng similarity làm điểm
-phát âm chính thức. Các API realtime/upload dưới đây giữ nguyên.
+In an async file-selection/button handler, call assessPronunciation(file, 'The cat is sitting on the mat.', controller.signal), then describeAccuracy(result). Create controller with new AbortController(); call controller.abort() when cancelling or leaving the screen. Keep a loading state, disable duplicate submissions and catch errors. Cancellation does not refund quota or guarantee that server inference stops.
 
-## Kết nối tự động, không cấu hình URL/token
+For an explicit single-word pronunciation, pass expectedPhonemes: ['k', 'æ', 't'] with referenceText: 'cat'. Each entry must be a complete model vocabulary token. Do not split IPA into characters, include enclosing slashes or reuse these tokens for a different passage. Normally omit expectedPhonemes for sentences; server-side text-to-phoneme support must be installed.
+
+## Pronunciation inputs and results
+
+Only en-US pronunciation is supported, independently of transcription language. Audio must be nonempty, at most 120 seconds and 25 MiB. Reference text must contain 1–4096 characters. The service validates duration and supported formats; WAV PCM16 mono at 16 kHz is suitable. The default assessment request timeout is 11 minutes. Each assessment obtains its own session and consumes quota; there is no automatic retry.
+
+| Result field | Meaning |
+| --- | --- |
+| overall | Versioned overall local score, status, scale, formula, components and availability reason |
+| reference_text, expected_phonemes | Passage and expected model tokens |
+| observed_phonemes | Recognized tokens with approximate start/end seconds |
+| comparisons | match/substitution/deletion/insertion; missing sounds can have null times |
+| phoneme_similarity, phoneme_error_rate | Edit-based similarity and error rate, not calibrated grades |
+| breakdown | Optional criteria, phoneme error counts, acoustic pauses and limitations |
+| acoustic_frames, psd_frequencies_hz | Formants, pitch, intensity and spectral measurements |
+| warnings, calibrated, timing_method | Interpretation limits; calibration is false and timing is approximate |
+
+Older services may omit overall and breakdown or return null. Show unavailable criteria as "Unavailable", never zero. Display warnings, criterion method/reason and breakdown.limitations. Error counts refer to phonemes, not words.
+
+Accuracy = 100*M/(M+S+I); completeness = 100*(N-D)/N, where M/S/D/I mean matches/substitutions/omissions/insertions and N is reference token count. Substitutions count as covered for completeness. Similarity = 100*max(0, 1-(S+D+I)/N), so it differs from accuracy. Fluency is an energy-continuity proxy and can be inflated by noise. Prosody provides pitch measurements without a score. Overall pronunciation is the harmonic mean of Accuracy and Completeness. Vocabulary, grammar and topic scores are unavailable. These estimates are not calibrated educational grades, Azure scores or pass/fail decisions.
+
+## Realtime transcription and recorded audio
 
 ```ts
-import { createSpeakingService, RealtimeSpeakingClient, SpeakingMicrophone } from '@hoanghong/speaking-core';
+import {
+  createSpeakingService, RealtimeSpeakingClient, SpeakingMicrophone,
+} from '@hoanghong/speaking-core';
+
 const config = createSpeakingService();
 const client = new RealtimeSpeakingClient(config.realtime);
-const mic = new SpeakingMicrophone(client, config.microphone);
+const microphone = new SpeakingMicrophone(client, config.microphone);
 const unsubscribe = client.onEvent(event => {
   if (event.type === 'partial') console.log(event.text);
   if (event.type === 'clientError') console.error(event.error);
 });
-// Gọi từ handler nút bắt đầu; không tự ghi âm khi load trang:
-export async function start() { await mic.start({ language: 'vi' }); }
+
+// Call from explicit user actions; catch failures in the UI.
+export async function start() { await microphone.start({ language: 'en' }); }
 export async function stop() {
-  const result = await mic.stop();
+  const result = await microphone.stop();
   return { text: result.result?.text ?? '', audio: result.audio };
 }
-export async function cancel() { await mic.cancel(); }
-export async function dispose() { unsubscribe(); await mic.dispose(); }
+export async function cancel() { await microphone.cancel(); }
+export async function dispose() { unsubscribe(); await microphone.dispose(); }
 ```
 
-Audio là File WAV PCM16 mono 16 kHz, gồm phần audio cuối khi dừng. Dùng URL.createObjectURL để nghe/tải rồi revokeObjectURL; hoặc FormData để lưu. Cần browser AudioWorklet/getUserMedia và HTTPS/localhost. Public worklet được tải tự động; không copy asset. Gọi dispose ở lifecycle unmount.
+The returned audio is a WAV File, PCM16 mono at 16 kHz, including final buffered samples. Pass it to the pronunciation helper with the passage for a separate assessment. Recording/transcription and pronunciation each create a session. Realtime supports 300 seconds, but assessed recordings must stay within the pronunciation limit of 120 seconds.
 
-## Upload
+Microphone capture requires HTTPS or localhost, getUserMedia and AudioWorklet. The worklet loads automatically. Dispose when unmounting. For playback/download, create an object URL and revoke it when replacing audio or leaving. Use FormData for storage uploads; do not serialize audio into JSON.
+
+## Upload for transcription
 
 ```ts
 import { BatchSpeakingClient, createSpeakingService } from '@hoanghong/speaking-core';
-const batch = new BatchSpeakingClient(createSpeakingService().batch);
-export async function upload(file: File) {
-  if (file.size > 25 * 1024 * 1024) throw new Error('File tối đa 25 MiB');
-  let job = await batch.submitAudio(file, 'vi', crypto.randomUUID(), 'short_answer', file.name);
-  const deadline = Date.now() + 600000;
+
+export async function transcribe(file: File) {
+  if (!file.size || file.size > 25 * 1024 * 1024) {
+    throw new Error('Select nonempty audio of at most 25 MiB.');
+  }
+  const batch = new BatchSpeakingClient(createSpeakingService().batch);
+  let job = await batch.submitAudio(file, 'en', crypto.randomUUID(), 'short_answer', file.name);
+  const deadline = Date.now() + 600_000;
   while (!['succeeded', 'no_speech', 'failed', 'cancelled'].includes(job.status)) {
-    if (Date.now() > deadline) throw new Error('Hết thời gian chờ');
+    if (Date.now() > deadline) throw new Error('Transcription timed out.');
     await new Promise(resolve => setTimeout(resolve, 1500));
     job = await batch.getResult(job.job_id);
   }
   if (job.error) throw new Error(job.error.message);
-  if (job.status === 'cancelled') throw new Error('Đã hủy');
-  return { text: job.result?.text ?? '', audio: file, language: 'vi', serviceType: 'upload' };
+  if (job.status === 'cancelled') throw new Error('Transcription was cancelled.');
+  return { text: job.result?.text ?? '', audio: file };
 }
 ```
 
-Public client quản lý capability cho một job tại một thời điểm; tạo client/config riêng nếu xử lý nhiều job song song. Giữ client khi cần getResult/cancelJob cho job đó. Không chia sẻ capability hay serialize audio vào JSON.
+Retain the batch client for polling/cancellation. Each public batch configuration owns one job capability at a time; use separate clients/configurations for concurrent jobs. Audio is not automatically stored permanently.
 
-Ngôn ngữ en/ja/ko/zh/vi. Public service mặc định 5 phiên/IP/ngày, 100 phiên/ngày/instance (UTC); có thể được FSEL điều chỉnh. Mỗi phiên mới tính lượt kể cả khi lỗi; 429/QUOTA_EXCEEDED là hết lượt. SDK không tự retry/replay. Realtime tối đa 300 giây với buffer PCM 9,6 MB; dọn buffer khi lỗi/hủy. Upload không tự lưu vĩnh viễn file trên S3. Mạng/CSP cần cho phép truy cập dịch vụ FSEL. 404/503 có thể do server chưa triển khai public SDK/model. Cần server hỗ trợ bản 0.3 trước khi sử dụng.
+## Application keys and errors
 
-createSpeakingGateway và các constructor cấu hình thủ công cũ vẫn được giữ cho nhu cầu nâng cao; không cần dùng trong luồng public trên.
+Use createSpeakingService({ appKey: 'YOUR_APP_KEY' }) for an application key supplied by FSEL. It is a visible application identifier, not an administrative credential. Never embed service credentials. Without a key, the basic anonymous quota applies. Invalid/disabled keys do not fall back to anonymous access.
 
-## Key ứng dụng
+Transcription languages: en/ja/ko/zh/vi. Basic quota defaults to 5 sessions per IP/day and 100 per instance/day (UTC), subject to operator changes. Sessions count even when processing fails. Realtime PCM buffering is approximately 9.6 MB maximum. Network/CSP must permit SDK connections and worklet loading.
 
-Dùng `createSpeakingService({ appKey: "KEY_DUOC_FSEL_CAP" })` thay cho lời gọi không tham số. SDK gửi key khi cấp phiên; quota quản lý phía server, không có URL cần cấu hình. Key này là định danh ứng dụng có thể quan sát ở frontend, không có quyền quản trị. INVALID_APP_KEY/401 nghĩa là sai hoặc đã bị khóa; không tự fallback ẩn danh.
+Handle SpeakingError.code/status/message: INVALID_REFERENCE, INVALID_AUDIO, INVALID_APP_KEY/401, QUOTA_EXCEEDED/429, unsupported audio/415, validation/422, missing functionality/404 and unavailable model/service/503. Cancellation and timeouts also reject. Do not automatically retry or replay audio. Installing the library does not deploy or verify service/model readiness.
 
 
-## Application key
+## Overall pronunciation output (0.6.0)
 
-Pass the application key provided by FSEL as `appKey`. The SDK handles service
-connections automatically. Quotas and key status are managed by the service.
-App keys identify an application and are visible in browser/mobile applications.
-
-```ts
-createSpeakingService({ appKey: 'YOUR_APP_KEY' });
-```
+Use `result.overall?.score` for the overall chart.
+It is a local estimate, the harmonic mean of unrounded Accuracy and Completeness:
+`2*A*C/(A+C)`, or 0 if A+C=0. Fluency and Prosody are excluded. This is not an
+Azure or calibrated educational grade. `phoneme_similarity` is a separate legacy
+edit-distance diagnostic and can be zero while the overall score is positive.
 
+The overall object includes `version=local-accuracy-completeness-v1`, `scale=100`,
+`calibrated=false`, `status`, `method`, `reason`, `components` and
+`excluded_components` (Flutter: `excludedComponents`).
+Render null/unavailable scores as unavailable; preserve valid zero values.
+An older server can omit `overall`; the SDK never computes a replacement score.
+`breakdown.pronunciation` mirrors the overall criterion for compatibility.
+The server must be upgraded as well as the SDK.
 
-## Multi-criteria pronunciation diagnostics (0.5.0)
-
-Optional `breakdown` contains accuracy, completeness, fluency, prosody, pronunciation,
-vocabulary, grammar and topic. Each has score (0–100 or null), status
-(estimated/unavailable), method and reason. These are local diagnostics, not Azure
-scores or calibrated grades. Older services may omit breakdown.
-
-- Accuracy = 100 * matches / (matches + substitutions + insertions), at phoneme level.
-- Completeness = 100 * (reference phonemes - omissions) / reference phonemes.
-  Substitutions count as covered; this is coverage, not correctness.
-- Fluency is an energy continuity proxy = 100 * (1 - internal pause seconds / active span).
-  20 ms RMS windows, threshold=max(0.005, 0.1*peak RMS), internal gaps >=250 ms.
-  Excludes leading/trailing silence; requires >=2 s audio and >=1 s active span.
-  Noise may inflate the value and intentional pauses may lower it; this is not VAD/native fluency.
-- Prosody has no score. Pitch P90-P10 and standard deviation (semitones) are reported
-  with >=20 voiced frames and >=2 s audio. Variation does not imply correct intonation.
-- Overall pronunciation and content scores remain null: no calibrated composite,
-  stress model, transcript/grammar evaluator or topic evaluator is configured.
-
-errors counts phoneme matches/substitutions/omissions/insertions. signal.pauses are
-acoustic intervals, not linguistic unexpected/missing breaks. Consult limitations
-and version=local-diagnostics-v1 before interpreting estimates.
+```json
+{"overall":{"score":50,"status":"estimated","method":"accuracy_completeness_hmean: 2*A*C/(A+C); zero if A+C=0","reason":null,"version":"local-accuracy-completeness-v1","scale":100,"calibrated":false,"components":["accuracy","completeness"],"excluded_components":["fluency","prosody"]}}
+```

package/dist/pronunciation.d.ts

@@ -33,6 +33,14 @@ export interface PronunciationCriterion {
     method: string;
     reason: string | null;
 }
+/** Overall local accuracy/completeness harmonic mean. Use this for the overall chart, never phoneme_similarity. */
+export interface PronunciationOverall extends PronunciationCriterion {
+    version: 'local-accuracy-completeness-v1';
+    scale: 100;
+    calibrated: false;
+    components: ['accuracy', 'completeness'];
+    excluded_components: ['fluency', 'prosody'];
+}
 export interface PronunciationBreakdown {
     version: 'local-diagnostics-v1';
     scope: 'scripted_phoneme_diagnostics';
@@ -84,6 +92,8 @@ export interface PronunciationAssessment {
     warnings: string[];
     /** Optional for compatibility with older service versions. */
     breakdown?: PronunciationBreakdown | null;
+    /** Absent on older servers. Null score is unavailable; zero is a valid score. */
+    overall?: PronunciationOverall | null;
 }
 export interface AssessPronunciationOptions {
     referenceText: string;

package/dist/pronunciation.js

@@ -57,6 +57,21 @@ export class PronunciationClient {
                     throw new SpeakingError('INVALID_RESPONSE', 'Invalid pronunciation diagnostic breakdown');
                 }
             }
+            if (body.overall != null) {
+                const overall = body.overall;
+                if (overall.version !== 'local-accuracy-completeness-v1' || overall.scale !== 100 ||
+                    overall.calibrated !== false || typeof overall.method !== 'string' ||
+                    JSON.stringify(overall.components) !== JSON.stringify(['accuracy', 'completeness']) ||
+                    JSON.stringify(overall.excluded_components) !== JSON.stringify(['fluency', 'prosody']) ||
+                    (overall.reason !== null && typeof overall.reason !== 'string') ||
+                    (overall.status !== 'estimated' && overall.status !== 'unavailable') ||
+                    (overall.status === 'unavailable' ? overall.score !== null :
+                        typeof overall.score !== 'number' || !Number.isFinite(overall.score) || overall.score < 0 || overall.score > 100) ||
+                    (body.breakdown != null && (overall.score !== body.breakdown.pronunciation.score ||
+                        overall.status !== body.breakdown.pronunciation.status || overall.method !== body.breakdown.pronunciation.method))) {
+                    throw new SpeakingError('INVALID_RESPONSE', 'Invalid overall pronunciation score');
+                }
+            }
             return body;
         }
         finally {

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@hoanghong/speaking-core",
-  "version": "0.5.0",
-  "description": "Online speaking SDK: gateway configuration, realtime microphone and audio upload",
+  "version": "0.6.0",
+  "description": "TypeScript SDK for speech transcription, audio upload and versioned local pronunciation assessment with typed overall and component scores.",
   "type": "module",
   "main": "dist/index.js",
   "types": "dist/index.d.ts",
@@ -32,5 +32,11 @@
   "publishConfig": {
     "access": "public",
     "registry": "https://registry.npmjs.org/"
-  }
+  },
+  "keywords": [
+    "speech",
+    "pronunciation-assessment",
+    "speech-to-text",
+    "sdk"
+  ]
 }