@hoanghong/speaking-core 0.4.3 → 0.6.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (5) hide show
  1. package/CHANGELOG.md +11 -0
  2. package/README.md +131 -46
  3. package/dist/pronunciation.d.ts +53 -0
  4. package/dist/pronunciation.js +30 -0
  5. package/package.json +9 -3
package/CHANGELOG.md CHANGED
@@ -1,3 +1,14 @@
1
+ ## 0.6.0
2
+
3
+ - Versioned overall local pronunciation score (harmonic mean of accuracy and completeness).
4
+ - Explicit score scale, availability, components, exclusions and calibration status.
5
+ - New clients accept older servers without inventing an overall score.
6
+
7
+ ## 0.5.0
8
+
9
+ - Optional multi-criteria local pronunciation diagnostics with methods, limitations and unavailable scores.
10
+ - Compatible with older service responses.
11
+
1
12
  ## 0.4.3
2
13
 
3
14
  - Remove service addresses from public integration examples; no runtime changes.
package/README.md CHANGED
@@ -1,89 +1,174 @@
1
1
  # @hoanghong/speaking-core
2
2
 
3
- SDK online cho người tự làm UI. Bản test 0.4.3 cài bằng `npm install ./vendor/hoanghong-speaking-core-0.4.3.tgz` từ bộ ZIP, chưa publish registry. Angular dùng @hoanghong/speaking-angular với component có sẵn; Flutter dùng fsel_speaking. MIT.
3
+ TypeScript SDK for custom speaking interfaces: realtime transcription, audio upload and US English pronunciation diagnostics. Models run on the service. The SDK manages connections and short-lived sessions; no service URL or service token is needed. MIT license.
4
4
 
5
- ## Đánh giá phát âm tiếng Anh Mỹ
5
+ ## Install from npm
6
+
7
+ ```bash
8
+ npm install @hoanghong/speaking-core@0.6.0
9
+ ```
10
+
11
+ ## Install the supplied version 0.6.0 archive
12
+
13
+ Copy the tarball from the release ZIP into your application's vendor directory:
14
+
15
+ ```bash
16
+ npm install ./vendor/hoanghong-speaking-core-0.6.0.tgz
17
+ ```
18
+
19
+ These instructions use the supplied archive and do not assume registry publication. Angular applications may use @hoanghong/speaking-angular; Flutter applications may use fsel_speaking.
20
+
21
+ ## Assess pronunciation from a file
22
+
23
+ Ask the learner to read the exact reference passage. Pass a selected File or recorded Blob to assessAudio. Transcription alone does not assess pronunciation.
6
24
 
7
25
  ```ts
8
- import { PronunciationClient, createSpeakingService } from '@hoanghong/speaking-core';
9
- const pronunciation = new PronunciationClient(createSpeakingService().pronunciation);
10
- export async function assess(file: File) {
11
- return pronunciation.assessAudio(file, {
12
- referenceText: 'cat', expectedPhonemes: ['k', 'æ', 't'], filename: file.name,
13
- });
26
+ import {
27
+ createSpeakingService, PronunciationClient, PronunciationAssessment, SpeakingError,
28
+ } from '@hoanghong/speaking-core';
29
+
30
+ const config = createSpeakingService();
31
+ // If FSEL supplied an application key, use instead:
32
+ // const config = createSpeakingService({ appKey: 'YOUR_APP_KEY' });
33
+ const pronunciation = new PronunciationClient(config.pronunciation);
34
+
35
+ export async function assessPronunciation(
36
+ file: File,
37
+ referenceText: string,
38
+ signal?: AbortSignal,
39
+ ): Promise<PronunciationAssessment> {
40
+ try {
41
+ return await pronunciation.assessAudio(file, {
42
+ referenceText,
43
+ filename: file.name,
44
+ signal,
45
+ });
46
+ } catch (error) {
47
+ if (error instanceof SpeakingError) {
48
+ console.error(error.code, error.status, error.message);
49
+ }
50
+ throw error; // Let the UI display the error and offer a deliberate retry.
51
+ }
52
+ }
53
+
54
+ export function describeOverall(result: PronunciationAssessment): string {
55
+ const value = result.overall;
56
+ return value?.status === 'estimated' && value.score != null
57
+ ? `${value.score.toFixed(1)} / ${value.scale} (local estimate)`
58
+ : value?.reason ?? 'Unavailable: upgrade the server if needed';
59
+ }
60
+
61
+ export function describeAccuracy(result: PronunciationAssessment): string {
62
+ const accuracy = result.breakdown?.accuracy;
63
+ console.log(result.comparisons, result.warnings);
64
+ return accuracy?.status === 'estimated' && accuracy.score != null
65
+ ? `${accuracy.score.toFixed(1)} / 100 (estimate)`
66
+ : 'Unavailable';
14
67
  }
15
68
  ```
16
69
 
17
- Audio tối đa 120 giây/25 MiB, reference tối đa 4096 ký tự; timeout client 11 phút.
18
- Đọc cả đoạn: thay referenceText bằng đoạn mẫu; bỏ expectedPhonemes khi server có eSpeak.
19
- Các âm tùy chọn là token vocabulary, không tách IPA theo ký tự. Mỗi lần gọi cấp phiên riêng,
20
- không tự retry. AbortSignal được hỗ trợ. Server phải triển khai endpoint/model phát âm;
21
- bộ cài này chưa xác nhận hosted endpoint đã sẵn sàng. Gateway riêng dùng cấu hình
22
- `createSpeakingGateway(...).pronunciation`, giữ credential của service ở backend.
23
- Response typed PronunciationAssessment: âm mẫu/nhận dạng, comparisons, timestamp ước lượng,
24
- formant/pitch/intensity/PSD và warnings. `calibrated=false`, không dùng similarity làm điểm
25
- phát âm chính thức. Các API realtime/upload dưới đây giữ nguyên.
70
+ In an async file-selection/button handler, call assessPronunciation(file, 'The cat is sitting on the mat.', controller.signal), then describeAccuracy(result). Create controller with new AbortController(); call controller.abort() when cancelling or leaving the screen. Keep a loading state, disable duplicate submissions and catch errors. Cancellation does not refund quota or guarantee that server inference stops.
71
+
72
+ For an explicit single-word pronunciation, pass expectedPhonemes: ['k', 'æ', 't'] with referenceText: 'cat'. Each entry must be a complete model vocabulary token. Do not split IPA into characters, include enclosing slashes or reuse these tokens for a different passage. Normally omit expectedPhonemes for sentences; server-side text-to-phoneme support must be installed.
73
+
74
+ ## Pronunciation inputs and results
75
+
76
+ Only en-US pronunciation is supported, independently of transcription language. Audio must be nonempty, at most 120 seconds and 25 MiB. Reference text must contain 1–4096 characters. The service validates duration and supported formats; WAV PCM16 mono at 16 kHz is suitable. The default assessment request timeout is 11 minutes. Each assessment obtains its own session and consumes quota; there is no automatic retry.
77
+
78
+ | Result field | Meaning |
79
+ | --- | --- |
80
+ | overall | Versioned overall local score, status, scale, formula, components and availability reason |
81
+ | reference_text, expected_phonemes | Passage and expected model tokens |
82
+ | observed_phonemes | Recognized tokens with approximate start/end seconds |
83
+ | comparisons | match/substitution/deletion/insertion; missing sounds can have null times |
84
+ | phoneme_similarity, phoneme_error_rate | Edit-based similarity and error rate, not calibrated grades |
85
+ | breakdown | Optional criteria, phoneme error counts, acoustic pauses and limitations |
86
+ | acoustic_frames, psd_frequencies_hz | Formants, pitch, intensity and spectral measurements |
87
+ | warnings, calibrated, timing_method | Interpretation limits; calibration is false and timing is approximate |
88
+
89
+ Older services may omit overall and breakdown or return null. Show unavailable criteria as "Unavailable", never zero. Display warnings, criterion method/reason and breakdown.limitations. Error counts refer to phonemes, not words.
90
+
91
+ Accuracy = 100*M/(M+S+I); completeness = 100*(N-D)/N, where M/S/D/I mean matches/substitutions/omissions/insertions and N is reference token count. Substitutions count as covered for completeness. Similarity = 100*max(0, 1-(S+D+I)/N), so it differs from accuracy. Fluency is an energy-continuity proxy and can be inflated by noise. Prosody provides pitch measurements without a score. Overall pronunciation is the harmonic mean of Accuracy and Completeness. Vocabulary, grammar and topic scores are unavailable. These estimates are not calibrated educational grades, Azure scores or pass/fail decisions.
26
92
 
27
- ## Kết nối tự động, không cấu hình URL/token
93
+ ## Realtime transcription and recorded audio
28
94
 
29
95
  ```ts
30
- import { createSpeakingService, RealtimeSpeakingClient, SpeakingMicrophone } from '@hoanghong/speaking-core';
96
+ import {
97
+ createSpeakingService, RealtimeSpeakingClient, SpeakingMicrophone,
98
+ } from '@hoanghong/speaking-core';
99
+
31
100
  const config = createSpeakingService();
32
101
  const client = new RealtimeSpeakingClient(config.realtime);
33
- const mic = new SpeakingMicrophone(client, config.microphone);
102
+ const microphone = new SpeakingMicrophone(client, config.microphone);
34
103
  const unsubscribe = client.onEvent(event => {
35
104
  if (event.type === 'partial') console.log(event.text);
36
105
  if (event.type === 'clientError') console.error(event.error);
37
106
  });
38
- // Gọi từ handler nút bắt đầu; không tự ghi âm khi load trang:
39
- export async function start() { await mic.start({ language: 'vi' }); }
107
+
108
+ // Call from explicit user actions; catch failures in the UI.
109
+ export async function start() { await microphone.start({ language: 'en' }); }
40
110
  export async function stop() {
41
- const result = await mic.stop();
111
+ const result = await microphone.stop();
42
112
  return { text: result.result?.text ?? '', audio: result.audio };
43
113
  }
44
- export async function cancel() { await mic.cancel(); }
45
- export async function dispose() { unsubscribe(); await mic.dispose(); }
114
+ export async function cancel() { await microphone.cancel(); }
115
+ export async function dispose() { unsubscribe(); await microphone.dispose(); }
46
116
  ```
47
117
 
48
- Audio là File WAV PCM16 mono 16 kHz, gồm phần audio cuối khi dừng. Dùng URL.createObjectURL để nghe/tải rồi revokeObjectURL; hoặc FormData để lưu. Cần browser AudioWorklet/getUserMedia và HTTPS/localhost. Public worklet được tải tự động; không copy asset. Gọi dispose ở lifecycle unmount.
118
+ The returned audio is a WAV File, PCM16 mono at 16 kHz, including final buffered samples. Pass it to the pronunciation helper with the passage for a separate assessment. Recording/transcription and pronunciation each create a session. Realtime supports 300 seconds, but assessed recordings must stay within the pronunciation limit of 120 seconds.
49
119
 
50
- ## Upload
120
+ Microphone capture requires HTTPS or localhost, getUserMedia and AudioWorklet. The worklet loads automatically. Dispose when unmounting. For playback/download, create an object URL and revoke it when replacing audio or leaving. Use FormData for storage uploads; do not serialize audio into JSON.
121
+
122
+ ## Upload for transcription
51
123
 
52
124
  ```ts
53
125
  import { BatchSpeakingClient, createSpeakingService } from '@hoanghong/speaking-core';
54
- const batch = new BatchSpeakingClient(createSpeakingService().batch);
55
- export async function upload(file: File) {
56
- if (file.size > 25 * 1024 * 1024) throw new Error('File tối đa 25 MiB');
57
- let job = await batch.submitAudio(file, 'vi', crypto.randomUUID(), 'short_answer', file.name);
58
- const deadline = Date.now() + 600000;
126
+
127
+ export async function transcribe(file: File) {
128
+ if (!file.size || file.size > 25 * 1024 * 1024) {
129
+ throw new Error('Select nonempty audio of at most 25 MiB.');
130
+ }
131
+ const batch = new BatchSpeakingClient(createSpeakingService().batch);
132
+ let job = await batch.submitAudio(file, 'en', crypto.randomUUID(), 'short_answer', file.name);
133
+ const deadline = Date.now() + 600_000;
59
134
  while (!['succeeded', 'no_speech', 'failed', 'cancelled'].includes(job.status)) {
60
- if (Date.now() > deadline) throw new Error('Hết thời gian chờ');
135
+ if (Date.now() > deadline) throw new Error('Transcription timed out.');
61
136
  await new Promise(resolve => setTimeout(resolve, 1500));
62
137
  job = await batch.getResult(job.job_id);
63
138
  }
64
139
  if (job.error) throw new Error(job.error.message);
65
- if (job.status === 'cancelled') throw new Error('Đã hủy');
66
- return { text: job.result?.text ?? '', audio: file, language: 'vi', serviceType: 'upload' };
140
+ if (job.status === 'cancelled') throw new Error('Transcription was cancelled.');
141
+ return { text: job.result?.text ?? '', audio: file };
67
142
  }
68
143
  ```
69
144
 
70
- Public client quản lý capability cho một job tại một thời điểm; tạo client/config riêng nếu xử lý nhiều job song song. Giữ client khi cần getResult/cancelJob cho job đó. Không chia sẻ capability hay serialize audio vào JSON.
145
+ Retain the batch client for polling/cancellation. Each public batch configuration owns one job capability at a time; use separate clients/configurations for concurrent jobs. Audio is not automatically stored permanently.
71
146
 
72
- Ngôn ngữ en/ja/ko/zh/vi. Public service mặc định 5 phiên/IP/ngày, 100 phiên/ngày/instance (UTC); có thể được FSEL điều chỉnh. Mỗi phiên mới tính lượt kể cả khi lỗi; 429/QUOTA_EXCEEDED là hết lượt. SDK không tự retry/replay. Realtime tối đa 300 giây với buffer PCM 9,6 MB; dọn buffer khi lỗi/hủy. Upload không tự lưu vĩnh viễn file trên S3. Mạng/CSP cần cho phép truy cập dịch vụ FSEL. 404/503 có thể do server chưa triển khai public SDK/model. Cần server hỗ trợ bản 0.3 trước khi sử dụng.
147
+ ## Application keys and errors
73
148
 
74
- createSpeakingGateway và các constructor cấu hình thủ công cũ vẫn được giữ cho nhu cầu nâng cao; không cần dùng trong luồng public trên.
149
+ Use createSpeakingService({ appKey: 'YOUR_APP_KEY' }) for an application key supplied by FSEL. It is a visible application identifier, not an administrative credential. Never embed service credentials. Without a key, the basic anonymous quota applies. Invalid/disabled keys do not fall back to anonymous access.
75
150
 
76
- ## Key ứng dụng
151
+ Transcription languages: en/ja/ko/zh/vi. Basic quota defaults to 5 sessions per IP/day and 100 per instance/day (UTC), subject to operator changes. Sessions count even when processing fails. Realtime PCM buffering is approximately 9.6 MB maximum. Network/CSP must permit SDK connections and worklet loading.
77
152
 
78
- Dùng `createSpeakingService({ appKey: "KEY_DUOC_FSEL_CAP" })` thay cho lời gọi không tham số. SDK gửi key khi cấp phiên; quota quản lý phía server, không có URL cần cấu hình. Key này là định danh ứng dụng có thể quan sát ở frontend, không có quyền quản trị. INVALID_APP_KEY/401 nghĩa là sai hoặc đã bị khóa; không tự fallback ẩn danh.
153
+ Handle SpeakingError.code/status/message: INVALID_REFERENCE, INVALID_AUDIO, INVALID_APP_KEY/401, QUOTA_EXCEEDED/429, unsupported audio/415, validation/422, missing functionality/404 and unavailable model/service/503. Cancellation and timeouts also reject. Do not automatically retry or replay audio. Installing the library does not deploy or verify service/model readiness.
79
154
 
80
155
 
81
- ## Application key
156
+ ## Overall pronunciation output (0.6.0)
82
157
 
83
- Pass the application key provided by FSEL as `appKey`. The SDK handles service
84
- connections automatically. Quotas and key status are managed by the service.
85
- App keys identify an application and are visible in browser/mobile applications.
158
+ Use `result.overall?.score` for the overall chart.
159
+ It is a local estimate, the harmonic mean of unrounded Accuracy and Completeness:
160
+ `2*A*C/(A+C)`, or 0 if A+C=0. Fluency and Prosody are excluded. This is not an
161
+ Azure or calibrated educational grade. `phoneme_similarity` is a separate legacy
162
+ edit-distance diagnostic and can be zero while the overall score is positive.
86
163
 
87
- ```ts
88
- createSpeakingService({ appKey: 'YOUR_APP_KEY' });
164
+ The overall object includes `version=local-accuracy-completeness-v1`, `scale=100`,
165
+ `calibrated=false`, `status`, `method`, `reason`, `components` and
166
+ `excluded_components` (Flutter: `excludedComponents`).
167
+ Render null/unavailable scores as unavailable; preserve valid zero values.
168
+ An older server can omit `overall`; the SDK never computes a replacement score.
169
+ `breakdown.pronunciation` mirrors the overall criterion for compatibility.
170
+ The server must be upgraded as well as the SDK.
171
+
172
+ ```json
173
+ {"overall":{"score":50,"status":"estimated","method":"accuracy_completeness_hmean: 2*A*C/(A+C); zero if A+C=0","reason":null,"version":"local-accuracy-completeness-v1","scale":100,"calibrated":false,"components":["accuracy","completeness"],"excluded_components":["fluency","prosody"]}}
89
174
  ```
package/dist/pronunciation.d.ts CHANGED
@@ -26,6 +26,55 @@ export interface PronunciationAcousticFrame {
26
26
  intensity_db: number | null;
27
27
  psd_power_per_hz: number[];
28
28
  }
29
+ /** Local estimates, not Azure-equivalent or calibrated educational scores. */
30
+ export interface PronunciationCriterion {
31
+ score: number | null;
32
+ status: 'estimated' | 'unavailable';
33
+ method: string;
34
+ reason: string | null;
35
+ }
36
+ /** Overall local accuracy/completeness harmonic mean. Use this for the overall chart, never phoneme_similarity. */
37
+ export interface PronunciationOverall extends PronunciationCriterion {
38
+ version: 'local-accuracy-completeness-v1';
39
+ scale: 100;
40
+ calibrated: false;
41
+ components: ['accuracy', 'completeness'];
42
+ excluded_components: ['fluency', 'prosody'];
43
+ }
44
+ export interface PronunciationBreakdown {
45
+ version: 'local-diagnostics-v1';
46
+ scope: 'scripted_phoneme_diagnostics';
47
+ calibrated: false;
48
+ accuracy: PronunciationCriterion;
49
+ completeness: PronunciationCriterion;
50
+ fluency: PronunciationCriterion;
51
+ prosody: PronunciationCriterion;
52
+ pronunciation: PronunciationCriterion;
53
+ vocabulary: PronunciationCriterion;
54
+ grammar: PronunciationCriterion;
55
+ topic: PronunciationCriterion;
56
+ errors: {
57
+ matches: number;
58
+ substitutions: number;
59
+ omissions: number;
60
+ insertions: number;
61
+ };
62
+ signal: {
63
+ speech_span_seconds: number;
64
+ internal_pause_seconds: number;
65
+ pause_ratio: number | null;
66
+ pauses: Array<{
67
+ start: number;
68
+ end: number;
69
+ duration: number;
70
+ }>;
71
+ energy_threshold: number;
72
+ pitch_range_semitones: number | null;
73
+ pitch_std_semitones: number | null;
74
+ voiced_frames: number;
75
+ };
76
+ limitations: string[];
77
+ }
29
78
  export interface PronunciationAssessment {
30
79
  reference_text: string;
31
80
  language: 'en-US';
@@ -41,6 +90,10 @@ export interface PronunciationAssessment {
41
90
  psd_frequencies_hz: number[];
42
91
  model: string;
43
92
  warnings: string[];
93
+ /** Optional for compatibility with older service versions. */
94
+ breakdown?: PronunciationBreakdown | null;
95
+ /** Absent on older servers. Null score is unavailable; zero is a valid score. */
96
+ overall?: PronunciationOverall | null;
44
97
  }
45
98
  export interface AssessPronunciationOptions {
46
99
  referenceText: string;
package/dist/pronunciation.js CHANGED
@@ -42,6 +42,36 @@ export class PronunciationClient {
42
42
  typeof body.phoneme_similarity !== 'number' || !Array.isArray(body.comparisons)) {
43
43
  throw new SpeakingError('INVALID_RESPONSE', 'Invalid pronunciation assessment response');
44
44
  }
45
+ if (body.breakdown != null) {
46
+ const breakdown = body.breakdown;
47
+ const names = ['accuracy', 'completeness', 'fluency', 'prosody', 'pronunciation', 'vocabulary', 'grammar', 'topic'];
48
+ if (breakdown.version !== 'local-diagnostics-v1' || breakdown.calibrated !== false ||
49
+ breakdown.scope !== 'scripted_phoneme_diagnostics' || !breakdown.signal || !breakdown.errors ||
50
+ !Array.isArray(breakdown.limitations) || names.some(name => {
51
+ const criterion = breakdown[name];
52
+ return !criterion || typeof criterion.method !== 'string' ||
53
+ (criterion.status !== 'estimated' && criterion.status !== 'unavailable') ||
54
+ (criterion.status === 'unavailable' ? criterion.score !== null :
55
+ typeof criterion.score !== 'number' || !Number.isFinite(criterion.score) || criterion.score < 0 || criterion.score > 100);
56
+ })) {
57
+ throw new SpeakingError('INVALID_RESPONSE', 'Invalid pronunciation diagnostic breakdown');
58
+ }
59
+ }
60
+ if (body.overall != null) {
61
+ const overall = body.overall;
62
+ if (overall.version !== 'local-accuracy-completeness-v1' || overall.scale !== 100 ||
63
+ overall.calibrated !== false || typeof overall.method !== 'string' ||
64
+ JSON.stringify(overall.components) !== JSON.stringify(['accuracy', 'completeness']) ||
65
+ JSON.stringify(overall.excluded_components) !== JSON.stringify(['fluency', 'prosody']) ||
66
+ (overall.reason !== null && typeof overall.reason !== 'string') ||
67
+ (overall.status !== 'estimated' && overall.status !== 'unavailable') ||
68
+ (overall.status === 'unavailable' ? overall.score !== null :
69
+ typeof overall.score !== 'number' || !Number.isFinite(overall.score) || overall.score < 0 || overall.score > 100) ||
70
+ (body.breakdown != null && (overall.score !== body.breakdown.pronunciation.score ||
71
+ overall.status !== body.breakdown.pronunciation.status || overall.method !== body.breakdown.pronunciation.method))) {
72
+ throw new SpeakingError('INVALID_RESPONSE', 'Invalid overall pronunciation score');
73
+ }
74
+ }
45
75
  return body;
46
76
  }
47
77
  finally {
package/package.json CHANGED
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "name": "@hoanghong/speaking-core",
3
- "version": "0.4.3",
4
- "description": "Online speaking SDK: gateway configuration, realtime microphone and audio upload",
3
+ "version": "0.6.0",
4
+ "description": "TypeScript SDK for speech transcription, audio upload and versioned local pronunciation assessment with typed overall and component scores.",
5
5
  "type": "module",
6
6
  "main": "dist/index.js",
7
7
  "types": "dist/index.d.ts",
@@ -32,5 +32,11 @@
32
32
  "publishConfig": {
33
33
  "access": "public",
34
34
  "registry": "https://registry.npmjs.org/"
35
- }
35
+ },
36
+ "keywords": [
37
+ "speech",
38
+ "pronunciation-assessment",
39
+ "speech-to-text",
40
+ "sdk"
41
+ ]
36
42
  }