@fonoster/autopilot 0.7.4 → 0.7.5

package/README.md

@@ -1,3 +1,157 @@
-<a href="https://gitpod.io/#https://github.com/fonoster/fonoster"> <img src="https://img.shields.io/badge/Contribute%20with-Gitpod-908a85?logo=gitpod" alt="Contribute with Gitpod" />
+autopilot
+=================
+
+[![Autopilot](https://img.shields.io/badge/autopilot-api-brightgreen.svg)](https://fonoster.com)
+[![Version](https://img.shields.io/npm/v/@fonoster/autopilot.svg)](https://npmjs.org/package/@fonoster/autopilot)
+[![Downloads/week](https://img.shields.io/npm/dw/@fonoster/autopilot.svg)](https://npmjs.org/package/@fonoster/autopilot)
+[![License](https://img.shields.io/npm/l/@fonoster/autopilot.svg)](https://github.com/fonoster/fonoster/blob/main/package.json)
+
+The autopilot module provides a way to create Voice AI applications in Fonoster. It uses Fonoster internal Voice API and the power of Large Language Models to create a conversational experience with the calling party.
+
+* [Installation](#installation)
+* [Example](#example)
+* [Adding Knowledge Base](#configuring-the-knowledge-base)
+* [Adding Tools](#configuring-the-tools)
+* [What's Next](#whats-next)
+
+> [!WARNING]  
+> This is an overview of Fonoster's Autopilot. This module is still under heavy development, and the API is subject to change. This overview implements OpenAI models, but we are working to bring support for other providers like Groq and others.
+
+## Installation
+
+```sh-session
+$ npm install -g @fonoster/autopilot
+```
+
+The previous command will install the module globally.
+
+## Example
+
+To function, Autopilot needs a configuration file, an OpenAI API key, and an active Fonoster account or self-hosted Fonoster instance. 
+
+The configuration file has two major sections: `conversationSettings` and `languageModel`. The `conversationSettings` section contains the settings for the conversation, like the first message, the system template, and the transfer options. The `languageModel` section contains the settings for the language model, like the provider, the model, and the temperature, among others.
+
+```json
+{
+  "conversationSettings": {
+    "firstMessage": "Hello, this is Olivia from Dr. Green's Family Medicine. How can I assist you today?",
+    "systemTemplate": "You are a Customer Service Representative. You are here to help the caller with their needs.",
+    "systemErrorMessage": "I'm sorry, but I seem to be having trouble. Please try again later.",
+    "initialDtmf": "6589",
+    "transferOptions": {
+      "enabled": true,
+      "phoneNumber": "+15555555555",
+      "message": "Please hold while I transfer you to a live agent."
+    },
+    "idleOptions": {
+      "enabled": true,
+      "message": "Are you still there?",
+      "timeout": 10000,
+      "maxTimeoutCount": 3
+    }
+    },
+  "languageModel": {
+    "provider": "openai",
+    "model": "gpt-4o-mini",
+    "maxTokens": 250,
+    "temperature": 0.7,
+    "knowledgeBase": [],
+    "tools": []
+  }
+}
+```
+
+To run the Autopilot, you can use the following command:
+
+```sh-session
+$ ASSISTANTS=$(pwd)/assistants.json OPEN_API_KEY=your-ley autopilot
+```
+
+You will see a message like this:
+
+```sh-session
+2024-09-06 11:32:16.584 [info]: (voice) started voice server @ 0.0.0.0, port=50061 {}
+```
+
+This means that the Autopilot is running and ready to take calls. 
+
+Since you are running Autopilot locally, you must expose port 50061 to the Internet. To expose the port, you can use a service like [ngrok](https://ngrok.com/) 
+
+```sh-session
+$ ngrok tcp 50061
+```
+
+Take the endpoint provided by ngrok and configure it in your Fonoster account. You can find more information about how to do this in the [Fonoster documentation](https://fonoster.com/docs).
+
+## Language Model Providers
+
+The Autopilot supports multiple language model providers. The following is a list of the supported providers:
+
+| Provider   | Description                                                | Supported models
+|------------|------------------------------------------------------------|------------------------------------------------------------------------------|
+| OpenAI     | OpenAI provides various GPT models for conversational AI   | `gpt-4o`, `gpt-40-mini` |
+| Groq       | Groq offers high-performance AI models optimized for speed | `gemm-7b-it`, `llama3-groq-70b-8192-tool-use-preview`, `llama3-1-8b-instant` |
+
+## Adding Knowledge Base
+
+Coming soon...
+
+## Adding Tools
+
+A tool in Autopilot is a way to interact with external services to get information or perform actions. For example, you can use a tool to get the weather, the latest news, or an SMS.
+
+You can configure a new tool by adding a new entry in the `tools` array in the configuration file. The following is an example tool that gets the available appointment times for a specific date.
+
+```json
+{
+  "conversationSettings": { ... },
+  "languageModel": {
+    "provider": "openai",
+    "model": "gpt-4o-mini",
+    "maxTokens": 250,
+    "temperature": 0.7,
+    "knowledgeBase": [],
+    "tools": [
+      {
+        "name": "getAvailableTimes",
+        "description": "Get available appointment times for a specific date.",
+        "parameters": {
+          "type": "object",
+          "properties": {
+            "date": {
+              "type": "string",
+              "format": "date"
+            }
+          },
+          "required": [
+            "date"
+          ]
+        },
+        "operation": {
+          "type": "get",
+          "url": "https://api.example.com/appointment-times"
+        }
+      }
+    ]
+  }
+}
+```
+
+In addition to the `get` operation type, you can use the `post` operation type. The `post` operation type is used when sending data to the tool. When sending a post, you can optionally set `waitForResponse` to false, which will "fire and forget" the request. The default behavior is to wait for the response.
+
+The expected format for the response is a JSON object with the following structure:
+
+```json
+{ "result": "text to pass to the language model" }
+```
+
+For example:
+
+```json
+{ "result": "The available appointment times are 9:00 AM, 10:00 AM, and 11:00 AM." }
+```
+
+## What's Next
+
+The Autopilot is still under heavy development. The next steps are to add support for Retrieval-Augmented Generation(RAG), improve the language model integration, and improve the Finite State Machine (FSM) that powers the conversations. If you have any suggestions or feedback, please let us know.
 
-This module is part of the [Fonoster](https://fonoster.com) project. By itself, it does not do much. It is intended to be used as a dependency for other modules. For more information about the project, please visit [https://github.com/fonoster/fonoster](https://github.com/fonoster/fonoster).

package/dist/Autopilot.d.ts

@@ -1,13 +1,9 @@
-import { AutopilotConfig } from "./types";
+import { AutopilotParams } from "./types";
 declare class Autopilot {
-    private config;
-    private assistant;
+    private params;
     private actor;
-    private voice;
-    constructor(config: AutopilotConfig);
+    constructor(params: AutopilotParams);
     start(): void;
-    private createActor;
-    private subscribeToActorState;
     private setupVoiceStream;
     private handleVoicePayload;
     private setupSpeechGathering;

package/dist/Autopilot.js

@@ -19,55 +19,40 @@ exports.Autopilot = void 0;
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
-const common_1 = require("@fonoster/common");
 const logger_1 = require("@fonoster/logger");
 const xstate_1 = require("xstate");
-const assistants_1 = require("./assistants");
 const machine_1 = require("./machine/machine");
-const vad_1 = require("./vad");
 const logger = (0, logger_1.getLogger)({ service: "autopilot", filePath: __filename });
 class Autopilot {
-    constructor(config) {
-        this.config = config;
-        this.assistant = (0, assistants_1.makeAssistant)(config.assistantConfig);
-        this.actor = this.createActor();
-        this.voice = config.voice;
-    }
-    start() {
-        this.actor.start();
-        this.setupSpeechGathering();
-        this.setupVoiceStream();
-        this.subscribeToActorState();
-    }
-    createActor() {
-        const { voice } = this.config;
-        return (0, xstate_1.createActor)(machine_1.machine, {
+    constructor(params) {
+        this.params = params;
+        const { voice, languageModel, conversationSettings } = this.params;
+        this.actor = (0, xstate_1.createActor)(machine_1.machine, {
             input: {
-                ...this.config.assistantConfig,
-                assistant: this.assistant,
+                conversationSettings,
+                languageModel: languageModel,
                 voice
             }
         });
     }
-    subscribeToActorState() {
+    start() {
+        this.actor.start();
         this.actor.subscribe((state) => {
             logger.verbose("actor's new state is", { state: state.value });
         });
+        this.setupVoiceStream();
+        this.setupSpeechGathering();
     }
     async setupVoiceStream() {
-        const stream = await this.config.voice.stream({
-            direction: common_1.StreamDirection.OUT
-        });
-        const vad = await (0, vad_1.makeVad)();
-        stream.onPayload(this.handleVoicePayload(vad));
+        const { voice, vad } = this.params;
+        const stream = await voice.stream();
+        stream.onData(this.handleVoicePayload(vad));
     }
     handleVoicePayload(vad) {
-        return async (payload) => {
+        return (chunk) => {
             try {
-                // TODO: Investigate why we need to cast this to Float32Array
-                const data = payload.data;
-                await vad(data, (event) => {
-                    if (event === "SPEECH_START" || event === "SPEECH_END") {
+                vad.processChunk(chunk, (event) => {
+                    if (["SPEECH_START", "SPEECH_END"].includes(event)) {
                         logger.verbose("received speech event", { event });
                         this.actor.send({ type: event });
                     }
@@ -79,11 +64,9 @@ class Autopilot {
         };
     }
     async setupSpeechGathering() {
-        const stream = await this.voice.sgather({
-            source: common_1.StreamGatherSource.SPEECH
-        });
-        stream.onPayload((payload) => {
-            const { speech } = payload;
+        const { voice } = this.params;
+        const stream = await voice.sgather();
+        stream.onData((speech) => {
             logger.verbose("received speech result", { speech });
             if (speech) {
                 this.actor.send({ type: "SPEECH_RESULT", speech });