testaugnitorecorder4 1.0.46

package/README.md

@@ -0,0 +1,179 @@
+# Async JS Client for Voice Backend
+
+**Contents**
+
+- [Intent](#intent)
+- [Components](#components)
+- [Usage](#usage)
+- [Running an Example](#running-an-example)
+
+## Intent
+
+The rationale behind an async implementation of the JS voice client is to decouple the following sets of tasks...
+
+- generating an audio stream
+- processing this audio stream (resample, conversion, etc.)
+- streaming to server over a websocket connection
+- handling ASR returned from server
+
+...such that each of the above run independently in their own thread insulated from each other and communicate via message passing.
+
+## Components
+
+The async JS client is composed of the following components, each portraying well-defined and exclusive roles.
+
+### Streamer
+
+Streamer is the entrypoint of the async JS client. It's role is two-fold:
+
+- initialise the **Worklet** and the **Executor** components
+- manage communication and control between all components
+
+When a new dictation session is started by the end-user, following steps are executed:
+
+1. Streamer is initialised along with the Worklet and the Executor
+2. It initialises the audio context post which audio generation begins
+3. It sends the audio packets to the Worklet for further processing
+4. It receives the processed and buffered packets from the Worklet and sends it to the Executor for ASR reception, processing and presentation.
+5. It repeats again from (3.)
+
+When a dictation session is stopped by the end-user, following steps are executed:
+
+1. Streamer stops the audio context
+2. It sends DONE messages to Worklet and Executor asking them to close gracefully
+
+_NOTE: Audio source in the existing implementation is in the form of a recorded audio played in a loop. To support audio generation through a microphone, please uncomment pertinent code in Streamer._
+
+### Worklet
+
+Worklet is employed mainly for audio processing, employing [AudioWorklets](https://developer.mozilla.org/en-US/docs/Web/API/AudioWorklet). In this context, it can be used to resample the incoming audio stream before sending the same to the server. It also keeps the processed audio packets buffered for a short period of time before sending it back to the Streamer to decrease overhead of message passing between OS-level threads running itself and the Streamer.
+
+### Executor
+
+This component deals with the following tasks:
+
+- obtain processed audio packets from the Streamer and stream it to the server over a websocket connection
+- obtain ASR from the server, process it well before pasting it to an editor screen
+
+Executor manages a websocket connection with the server in a [Web Worker](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers). Audio packets received from the Streamer are buffered in a read-queue to be sent over to the server. It houses 3 daemons running periodically:
+
+1. **Consumer**: It consumes audio packets from the read-queue and streams it to the server over a websocket connection
+2. **Healthcheck**: It oversees the websocket connection and closes it forcefully when the server is unreachable.
+3. **IdleThread**: It keeps track of the fact that data is sent regularly over the websocket connection. If the connection remains idle for a given period of time, this daemon closes the connection gracefully thus freeing up network and server resources.
+
+## Usage
+
+### Initialise Audio Stream
+
+This can be employed when a new dictation session begins by the user with an intent of creating a clinical document.
+
+```js
+const recorderInstance = new AugnitoRecorder(
+  {
+    serverURL: WS_URL,
+    enableLogs: false,
+    isDebug: false,
+    bufferInterval: 1,
+    EOS_Message: "EOS",
+    socketTimeoutInterval: 10000,
+    shouldSendAudioDataSequence: false,
+  },
+  heavyOp
+);
+recorderInstance.toggleStartStopAudioStream(); //if you want to start/stop audio on button click
+or;
+recorderInstance.togglePauseResumeAudioStream(); //if you want to pause/resume audio on button click
+```
+
+- `WS_URL` is the server websocket endpoint to which the client connects to, to stream audio.
+- `heavyOp` is a CPU-intensive operation which is run on the received ASR before displaying it on the editor. A dummy example is as follows:
+- `enableLogs` is set to true if you want to see logs in console else set to false
+- `isDebug` is set to true if you want to save recorded audio to a file else set to false
+- `bufferInterval` is set to any non negative integer, indicates the buffer size interval in seconds, default value is 1 sec
+- `EOS_Message` is set to any string that indicates EOS or can be undefined, default value is EOS
+- `socketTimeoutInterval` is set to any socket timeout interval as desired, default value is 10000
+- `shouldSendAudioDataSequence` is set to true if you want the audio packet sequence sent with every packet as header else set to false
+
+```js
+const heavyOp = (text) => {
+  let num = -1;
+  let iters = 0;
+  let sum = 0;
+  for (let i = 0; i < text.length; i++) {
+    sum += text.charCodeAt(i);
+  }
+  console.debug(`Running Heavy Operation for "${text}" with sum: ${sum}`);
+  while (num != sum) {
+    num = Math.floor(Math.random() * 100000);
+    iters++;
+  }
+  console.log(`Iterations completed for sum(${sum}): ${iters}`);
+  return text;
+};
+```
+
+### Audio Stream Pause and Resume
+
+This can be employed when the user pauses and resumes the dictation session emulating a toggle of the microphone button.
+
+```js
+recorderInstance.pauseAudio();
+
+/*
+Do something interesting here...
+*/
+
+recorderInstance.resumeAudio();
+```
+
+### Stop Audio Stream
+
+This can be employed when the user has completed their dictation session after finishing off compiling the medical document.
+
+```js
+recorderInstance.stopAudio();
+```
+
+### Library Callbacks
+
+These callbacks can be employed to read socket connection state change event, session event, speech text output event or any other error event from the library.
+
+```js
+recorderInstance.onStateChanged = (connected) => {
+  //connected is true when socket is opened and false when socket is closed
+};
+recorderInstance.onSessionEvent = (response) => {
+  //handle all meta events
+};
+recorderInstance.onError = (error) => {
+  //handle any error message
+};
+recorderIns.onPartialResult = (text) => {
+  // hypothesis text output generated as the user speaks
+};
+recorderInstance.onFinalResult = (textResponse) => {
+  //text reponse can be parsed to read the speech output json
+};
+```
+
+## Compile library
+
+Below steps to compile library and the output is generated in dist folder
+
+```sh
+cd AugnitoRecorderJS
+npm run build
+```
+
+## Running an Example
+
+```sh
+cd AugnitoRecorderJS
+python3 -m http.server
+```
+
+Once done, fire up the webpage at: http://localhost:8000. The webpage has following controls for speech:
+
+- **Start Stream**: This begins playing the recorded audio which is streamed to the server and the ASR obtained pertaining to it is pasted on screen live. This control marks the start of a new session with the server and so a new WS connection is created for audio streaming and ASR reception. The recording and the ensuing ASR can be paused and resumed using the same control.
+- **Stop Stream**: This stops the recorded audio from playing marking the end of the client session. The connection with the server is severed when this control is engaged.
+- **Go Crazy...**: This control randomly starts, pauses, resumes and stops the audio streaming session in a loop. It employs the above two controls as it's building block and is a hands-free approach to communicate with the server in the form of a virtual user.

package/dist/augnitoRecorder.d.ts

@@ -0,0 +1,54 @@
+export declare class AugnitoRecorder {
+  private WebsocketURL;
+  private enableLogs;
+  private isDebug;
+  private streamer;
+  private heavyOp;
+  private bufferInterval;
+  private eosMessage;
+  private socketTimeoutInterval;
+  onSessionEvent: (data: any) => void;
+  onStateChanged: (isRecording: boolean) => void;
+  onError: (errorMessage: string) => void;
+  onPartialResult: (hype: string) => void;
+  onFinalResult: (recipe: any) => void;
+  onOtherResults: (message: string) => void;
+  onIntensity: (intenity: number) => void;
+  showLog: (event: string) => void;
+  constructor(
+    config: {
+      serverURL?: string;
+      enableLogs: boolean;
+      isDebug: boolean;
+      bufferInterval?: number;
+      pausedBufferInterval?: number;
+      EOS_Message?: string;
+      socketTimeoutInterval?: number;
+      shouldSendAudioDataSequence?: boolean;
+      shouldPreIntialiseRecorder?: boolean;
+      shouldReadIntensity?: boolean;
+      debounceDelay?: number;
+      switchToRegularSpeechProfile?: boolean
+    },
+    heavyOp?: any
+  );
+  togglePauseResumeAudioStream(
+    audioDuration?: number,
+    socketURL?: string
+  ): void;
+  toggleStartStopAudioStream(audioDuration?: number, socketURL?: string): void;
+  startAudio(): void;
+  pauseAudio(): void;
+  resumeAudio(): void;
+  stopAudio(shouldSendEOS?: boolean, forceStopForPausedState?: boolean): void;
+  getBlob(): Blob;
+  log(event: any): void;
+  dispose(): void;
+  onSessionEventCallback(data: any): void;
+  onStateChangedCallback(isRecording: any): void;
+  onErrorCallback(errorMessage: any): void;
+  onPartialResultCallback(hype: any): void;
+  onFinalResultCallback(recipe: any): void;
+  showLogCallback(event: any): void;
+  onIntensityCallback(intensity: number): void;
+}