llmasaservice-client 0.0.1 → 0.0.2

package/CHANGELOG.md

@@ -0,0 +1,7 @@
+# llmasaservice-client
+
+## 0.0.2
+
+### Patch Changes
+
+- Added documentation and making the customer optional for calls

package/README.md

@@ -1 +1,180 @@
-# LLMAsAService.io for React Clients
+# LLMAsAService.io for Client Side Code
+A product by CASEy, Inc ([heycasey.io](https://heycasey.io))
+
+## What is LLMAsAService.io?
+Implementing AI LLM features into application is relatively easy using a vendors API. When we were building our product ([heycasey.io](https://heycasey.io)) we had a number of reliability and financial risk issues. We solved these, and decided to make our platform for managing LLM's available as a service. 
+
+The main features are -
+- Streaming responses to give near immediate customer results and feedback
+- Abort and Cancellation of streaming responses
+- Multiple vendors with load-sharing and failover when (not if) they have outages
+- Response caching for identical requests withing a given timeframe
+- Secure storage of API keys from one portal
+- Customer token budgeting and management (give trial users an amount of tokens to start with) - 0 to inhibit new customers
+- Customer data tenancy - customer requests can be routed to certain LLM providers and regions based on their settings (Eg. EU customers)
+- PII redaction. Clever tokenization of common PII data so no LLM vendor gets private information. Those tokens are replaced on return to the customer so the who redaction is transparent.
+- Call analytics - request success or errors, count, and token usage
+- Call analytics - country and region for security analysis (why are we getting a heap of requests from North Korea?)
+- Load shedding - if traffic increases beyond allocated vendor capacity limits, some customers can be rejected based on their allocated "Tier"
+
+There are two parts of using this service. The control panel at ([app.llmasaservice.io](https://app.llmasaservice.io)), and this library for connecting our service to your client side application.
+
+Using all of your LLM services with the above features becomes as simple as -
+
+```typescript
+import { useLLM } from 'useLLM';
+
+...
+const {send, response, idle} = useLLM({project_id: "[your LLMAsAService project id]"}); // get the project_id from the Embed page in the control panel
+
+...
+const handleChatClick = () => {
+    send("What is 1+1="); // calls the LLMs for a streaming response
+};
+
+// the response is streamed back and can be shown where needed
+return (
+    <div>
+       <button onClick={handleChatClick} disabled={!idle}>
+         Call Chat
+       </button>
+    
+       <div>{response}</div>
+    </div>);
+```
+
+## Step 1 - Register for a LLMAsAService account
+1. Register at  ([app.llmasaservice.io](https://app.llmasaservice.io)) and confirm your email address
+2. You will be prompted after first login to create a project (and accept privacy and terms)
+
+## Step 2 - Create your LLM Service Providers
+You can create any number of LLM Vendor service endpoints. The active services will be load balanced and used for failover if any vendor has an outage or is usage limited. You will ideally have at least two active at all times for reliable failover. You can also create a Chaos Monkey service that will randomly fail calls to prove that failover is effectively working.
+
+The settings you need are the same settings you would pass to the vendor using their API. We make those calls on your behalf from our servers after confiming those requests are allowed.
+
+After logging into the control panel, choose LLMServices from the right panel menu
+
+![LLM Service Page](images\LLMServicesPage.png)
+
+1. Click the Add Service box to create your first or another LLM Service Endpoint
+
+![Add LLM Service Page](images\AddLLMService.png)
+
+2. Name and choose your Vendor. We will boilerplate that vendors endpoint URL, header and body as a starting point. These will vary based on your needs and the vendor chosen (we are documenting each vendor now, until then email us if you have difficulty). Tip: We use the streaming features for each vendor. So, make sure that the streaming options are enabled. For example, for OpenAI add these at the end of the rest of the body JSON:
+
+```javascript
+{
+    .... ,
+   "stream": true,
+   "stream_options": {"include_usage": true}
+}
+```
+
+3. The inputs are validated. Main requirements is that the header and body are validated JSON inputs, and quotes for the parameters names and values are necessary with no ending comma before end brace characters eg. 
+
+Good:
+```javascript
+{
+  "Content-Type": "application/json",
+  "Authorization": "Bearer {{API-KEY}}"
+}
+```
+
+Bad (no quotes on names, and trailing , before ending brace):
+```javascript
+{
+  Content-Type: "application/json",
+  Authorization: "Bearer {{API-KEY}}",
+}
+```
+4. The {{API-KEY}} should be used where the API key should go. DO NOT! DO NOT! DO NOT! hardcode your API keys into the LLM Service form. We will check that the placeholder is used.
+5. Click Save and Close. The next step is mandatory before these services will work.
+6. Click on the Edit button for the new service. The Add or Update API Key and Test buttons will now be enabled. Click **Add or Update API Key**
+7. We properly encrypt and save your API key (we cannot retrieve it for you, if lost, create a new key from your vendor). Get the API key from your LLM Vendor's developer control panel and paste it into the dialog box and click Save. (one of the advantages of using LLMAsAService is that safe key management is in one place, we found this convenient and safer than using command line tools and config files)
+
+![API Key dialog](images\APIKey.png)
+
+8. Make a test call. Click the **Test Call** button and confirm you get a response
+
+![Test Call](images\TestCall.png)
+
+Repeat those steps for your other providers and configurations (one n north america, one in the EU, one for Azure or Anthropic, etc.)
+
+## Step 3 - Add the useLLM to your project code
+
+To enable the LLM features in your application we provide a react HOC and a hook. These client side components connect your app to the LLMService backend.
+
+1. Import our client side library (UI components coming soon in a different package, this package has no dependencies and we want it that way)
+
+```command
+npm i llmasaservice-client
+```
+(or the yarn equivalent)
+
+2. Instantiate the hook and the HOC (optional)
+
+```typescript
+
+/** used without HOC **/
+import { useLLM } from 'useLLM';
+const {send, response, idle} = useLLM({project_id: "[your LLMAsAService project id]"}); // get the project_id from the Embed page in the control panel
+
+
+/** using the HOC **/
+  //Index.tsx or App.tsx or similar, enclose the App inside the LLMServiceProvider
+  import { LLMServiceProvider } from "useLLM";
+
+  <LLMServiceProvider project_id="[your LLMAsAService project id]">
+    <App />
+  </LLMServiceProvider>
+
+  // inside your component pages
+  import { useLLM } from 'useLLM';
+  
+  const {send, response, idle} = useLLM();
+
+```
+
+3. Pass the customer making the call. If you wan to track and grant tokens to certain customers, you can pass them by a unique key (you choose, but must be encodedable in JSON, we use a UUID) and an customer identifying name. Tip: pass the customer at the level you want to track. A company id will allow all users for that company to be controlled as a group. We also don't want any PII. So, don't use an email address, we don't need it, and its another siurce of PII data leakage neither of us want.
+
+```typescript
+
+/** used without HOC **/
+import { useLLM } from 'useLLM';
+const {send, response, idle} = useLLM(
+    {
+        project_id: "[your LLMAsAService project id]", // get this from the Embed page in the control panel
+        customer: {
+            customer_id: "[your unique customer identifier]", // don't use email please. 
+            customer_name: "[a way of humans identifying this customer in the control panel]" 
+        }
+    });
+
+
+/** using the HOC **/
+  //Index.tsx or App.tsx or similar
+  import { LLMServiceProvider } from "useLLM";
+
+  <LLMServiceProvider 
+        project_id ="[your LLMAsAService project id]", // get this from the Embed page in the control panel
+        customer = {
+            customer_id: "[your unique customer identifier]", // don't use email please. 
+            customer_name: "[a way of humans identifying this customer in the control panel]" 
+        }>
+    <App />
+  </LLMServiceProvider>
+
+  // inside your component pages
+  import { useLLM } from 'useLLM';
+  
+  const {send, response, idle} = useLLM();
+
+```
+
+## Step 4 - Adding Chat features to your application
+
+Calling **send** makes a secure call to LLMAsAService where a response is marshalled back from the providers. That response is in the **response** property. 
+
+We have pre-built UIs in the works, but for now, you can call send and display the response wherever needed. An additional property **idle"" can be used to disable the send buttons when a response is ongoing. It will be true when idle, false when busy.
+
+We also accept Abort functionality, and are in the process of documenting that now. If you need it email help@heycasey.io and we'll sort you out.

package/dist/index.d.mts

@@ -1,4 +1,4 @@
-import React from 'react';
+import React, { ReactNode } from 'react';
 
 type LLMAsAServiceCustomer = {
     customer_id: string;
@@ -8,19 +8,14 @@ type LLMAsAServiceCustomer = {
 };
 interface LLMServiceType {
     project_id: string | undefined;
-    customer: LLMAsAServiceCustomer;
+    customer?: LLMAsAServiceCustomer;
 }
 declare const LLMService: React.Context<LLMServiceType | undefined>;
+interface UserProviderProps {
+    children: ReactNode;
+    project_id: string | undefined;
+    customer?: LLMAsAServiceCustomer;
+}
+declare const LLMServiceProvider: React.FC<UserProviderProps>;
 
-declare const useLLM: (options?: LLMServiceType) => {
-    response: string;
-    send: (prompt: string, messages?: {
-        role: string;
-        content: string;
-    }[], stream?: boolean, abortController?: AbortController, service?: string | null) => Promise<ReadableStreamDefaultReader<any> | string | undefined>;
-    stop: (controller: AbortController | null) => void;
-    idle: boolean;
-    error: string;
-};
-
-export { LLMService as LLMAsAService, useLLM };
+export { LLMService, LLMServiceProvider, type LLMServiceType };

package/dist/index.d.ts

@@ -1,4 +1,4 @@
-import React from 'react';
+import React, { ReactNode } from 'react';
 
 type LLMAsAServiceCustomer = {
     customer_id: string;
@@ -8,19 +8,14 @@ type LLMAsAServiceCustomer = {
 };
 interface LLMServiceType {
     project_id: string | undefined;
-    customer: LLMAsAServiceCustomer;
+    customer?: LLMAsAServiceCustomer;
 }
 declare const LLMService: React.Context<LLMServiceType | undefined>;
+interface UserProviderProps {
+    children: ReactNode;
+    project_id: string | undefined;
+    customer?: LLMAsAServiceCustomer;
+}
+declare const LLMServiceProvider: React.FC<UserProviderProps>;
 
-declare const useLLM: (options?: LLMServiceType) => {
-    response: string;
-    send: (prompt: string, messages?: {
-        role: string;
-        content: string;
-    }[], stream?: boolean, abortController?: AbortController, service?: string | null) => Promise<ReadableStreamDefaultReader<any> | string | undefined>;
-    stop: (controller: AbortController | null) => void;
-    idle: boolean;
-    error: string;
-};
-
-export { LLMService as LLMAsAService, useLLM };
+export { LLMService, LLMServiceProvider, type LLMServiceType };

package/dist/index.js

@@ -26,32 +26,12 @@ var __toESM = (mod, isNodeMode, target) => (target = mod != null ? __create(__ge
   mod
 ));
 var __toCommonJS = (mod) => __copyProps(__defProp({}, "__esModule", { value: true }), mod);
-var __async = (__this, __arguments, generator) => {
-  return new Promise((resolve, reject) => {
-    var fulfilled = (value) => {
-      try {
-        step(generator.next(value));
-      } catch (e) {
-        reject(e);
-      }
-    };
-    var rejected = (value) => {
-      try {
-        step(generator.throw(value));
-      } catch (e) {
-        reject(e);
-      }
-    };
-    var step = (x) => x.done ? resolve(x.value) : Promise.resolve(x.value).then(fulfilled, rejected);
-    step((generator = generator.apply(__this, __arguments)).next());
-  });
-};
 
 // index.ts
 var useLLM_exports = {};
 __export(useLLM_exports, {
-  LLMAsAService: () => LLMAsAService_default,
-  useLLM: () => useLLM_default
+  LLMService: () => LLMService,
+  LLMServiceProvider: () => LLMServiceProvider
 });
 module.exports = __toCommonJS(useLLM_exports);
 
@@ -61,126 +41,15 @@ var import_react2 = require("react");
 // src/LLMAsAService.tsx
 var import_react = __toESM(require("react"));
 var LLMService = (0, import_react.createContext)(void 0);
-var LLMAsAService_default = LLMService;
-
-// src/useLLM.ts
-var useLLM = (options) => {
-  var _a;
-  const [response, setResponse] = (0, import_react2.useState)("");
-  const [idle, setIdle] = (0, import_react2.useState)(true);
-  const [error, setError] = (0, import_react2.useState)("");
-  const context = (_a = (0, import_react2.useContext)(LLMService)) != null ? _a : options;
-  if (!context) {
-    throw new Error(
-      "useLLM must be used within a LLMServiceProvider or constructed with options in your useLLM() call."
-    );
-  }
-  const stop = (controller) => {
-    if (controller) controller.abort();
-    setIdle(true);
-  };
-  function send(_0) {
-    return __async(this, arguments, function* (prompt, messages = [{ role: "system", content: "You are a useful assistant." }], stream = true, abortController = new AbortController(), service = null) {
-      var _a2, _b, _c;
-      setResponse("");
-      setIdle(false);
-      let errorInFetch = "";
-      const url = "https://chat.llmasaservice.io/";
-      const responseBody = JSON.stringify({
-        projectId: (_a2 = context == null ? void 0 : context.project_id) != null ? _a2 : "",
-        serviceId: service,
-        prompt,
-        messages,
-        customer: (_b = context == null ? void 0 : context.customer) != null ? _b : {}
-      });
-      const options2 = {
-        method: "POST",
-        signal: abortController.signal,
-        mode: "cors",
-        headers: {
-          "Content-Type": "text/plain"
-          //"x-Amz-Content-Sha256": sha256.create().update(responseBody).hex(),
-          //"x-Amz-Content-Sha256": "UNSIGNED-PAYLOAD",
-        },
-        body: responseBody
-      };
-      try {
-        const response2 = yield fetch(url, options2);
-        if (!response2.ok) {
-          errorInFetch = `Error: Network error for service. (${response2.status} ${response2.statusText})`;
-        } else {
-          const reader = (_c = response2 == null ? void 0 : response2.body) == null ? void 0 : _c.getReader();
-          const decoder = new TextDecoder("utf-8");
-          setIdle(false);
-          if (!stream) {
-            setResponse(
-              yield readStream(reader, decoder, stream, {
-                signal: options2.signal
-              })
-            );
-          } else {
-            readStream(reader, decoder, stream, {
-              signal: options2.signal
-            });
-            return reader;
-          }
-        }
-      } catch (errorObject) {
-        errorInFetch = `Error: Having trouble connecting to chat service. (${errorObject.message})`;
-      }
-      if (errorInFetch !== "") {
-        setError(errorInFetch);
-        console.error(`Error: Error in fetch. (${errorInFetch})`);
-      }
-    });
-  }
-  function readStream(_0, _1) {
-    return __async(this, arguments, function* (reader, decoder, stream = true, { signal }) {
-      let errorInRead = "";
-      let result = "";
-      while (true) {
-        try {
-          if (signal.aborted) {
-            reader.cancel();
-            setIdle(true);
-            break;
-          }
-          const { value, done } = yield reader.read();
-          if (decoder.decode(value).startsWith("Error:")) {
-            errorInRead = decoder.decode(value).substring(6);
-            break;
-          }
-          if (done) {
-            setIdle(true);
-            break;
-          }
-          result += decoder.decode(value);
-          if (stream) setResponse((prevState) => result);
-        } catch (error2) {
-          if (error2.name === "AbortError") {
-            break;
-          }
-          errorInRead = `Reading error  ${error2.message}`;
-          break;
-        } finally {
-          if (signal.aborted) {
-            reader.releaseLock();
-          }
-        }
-      }
-      if (errorInRead !== "") {
-        setError(errorInRead);
-        reader.cancel();
-        setIdle(true);
-      }
-      return result;
-    });
-  }
-  return { response, send, stop, idle, error };
+var LLMServiceProvider = ({
+  children,
+  project_id,
+  customer
+}) => {
+  return /* @__PURE__ */ import_react.default.createElement(LLMService.Provider, { value: { project_id, customer } }, children);
 };
-var useLLM_default = useLLM;
 // Annotate the CommonJS export names for ESM import in node:
 0 && (module.exports = {
-  LLMAsAService,
-  useLLM
+  LLMService,
+  LLMServiceProvider
 });

package/dist/index.mjs

@@ -1,149 +1,17 @@
-var __async = (__this, __arguments, generator) => {
-  return new Promise((resolve, reject) => {
-    var fulfilled = (value) => {
-      try {
-        step(generator.next(value));
-      } catch (e) {
-        reject(e);
-      }
-    };
-    var rejected = (value) => {
-      try {
-        step(generator.throw(value));
-      } catch (e) {
-        reject(e);
-      }
-    };
-    var step = (x) => x.done ? resolve(x.value) : Promise.resolve(x.value).then(fulfilled, rejected);
-    step((generator = generator.apply(__this, __arguments)).next());
-  });
-};
-
 // src/useLLM.ts
 import { useContext, useState } from "react";
 
 // src/LLMAsAService.tsx
 import React, { createContext } from "react";
 var LLMService = createContext(void 0);
-var LLMAsAService_default = LLMService;
-
-// src/useLLM.ts
-var useLLM = (options) => {
-  var _a;
-  const [response, setResponse] = useState("");
-  const [idle, setIdle] = useState(true);
-  const [error, setError] = useState("");
-  const context = (_a = useContext(LLMService)) != null ? _a : options;
-  if (!context) {
-    throw new Error(
-      "useLLM must be used within a LLMServiceProvider or constructed with options in your useLLM() call."
-    );
-  }
-  const stop = (controller) => {
-    if (controller) controller.abort();
-    setIdle(true);
-  };
-  function send(_0) {
-    return __async(this, arguments, function* (prompt, messages = [{ role: "system", content: "You are a useful assistant." }], stream = true, abortController = new AbortController(), service = null) {
-      var _a2, _b, _c;
-      setResponse("");
-      setIdle(false);
-      let errorInFetch = "";
-      const url = "https://chat.llmasaservice.io/";
-      const responseBody = JSON.stringify({
-        projectId: (_a2 = context == null ? void 0 : context.project_id) != null ? _a2 : "",
-        serviceId: service,
-        prompt,
-        messages,
-        customer: (_b = context == null ? void 0 : context.customer) != null ? _b : {}
-      });
-      const options2 = {
-        method: "POST",
-        signal: abortController.signal,
-        mode: "cors",
-        headers: {
-          "Content-Type": "text/plain"
-          //"x-Amz-Content-Sha256": sha256.create().update(responseBody).hex(),
-          //"x-Amz-Content-Sha256": "UNSIGNED-PAYLOAD",
-        },
-        body: responseBody
-      };
-      try {
-        const response2 = yield fetch(url, options2);
-        if (!response2.ok) {
-          errorInFetch = `Error: Network error for service. (${response2.status} ${response2.statusText})`;
-        } else {
-          const reader = (_c = response2 == null ? void 0 : response2.body) == null ? void 0 : _c.getReader();
-          const decoder = new TextDecoder("utf-8");
-          setIdle(false);
-          if (!stream) {
-            setResponse(
-              yield readStream(reader, decoder, stream, {
-                signal: options2.signal
-              })
-            );
-          } else {
-            readStream(reader, decoder, stream, {
-              signal: options2.signal
-            });
-            return reader;
-          }
-        }
-      } catch (errorObject) {
-        errorInFetch = `Error: Having trouble connecting to chat service. (${errorObject.message})`;
-      }
-      if (errorInFetch !== "") {
-        setError(errorInFetch);
-        console.error(`Error: Error in fetch. (${errorInFetch})`);
-      }
-    });
-  }
-  function readStream(_0, _1) {
-    return __async(this, arguments, function* (reader, decoder, stream = true, { signal }) {
-      let errorInRead = "";
-      let result = "";
-      while (true) {
-        try {
-          if (signal.aborted) {
-            reader.cancel();
-            setIdle(true);
-            break;
-          }
-          const { value, done } = yield reader.read();
-          if (decoder.decode(value).startsWith("Error:")) {
-            errorInRead = decoder.decode(value).substring(6);
-            break;
-          }
-          if (done) {
-            setIdle(true);
-            break;
-          }
-          result += decoder.decode(value);
-          if (stream) setResponse((prevState) => result);
-        } catch (error2) {
-          if (error2.name === "AbortError") {
-            break;
-          }
-          errorInRead = `Reading error  ${error2.message}`;
-          break;
-        } finally {
-          if (signal.aborted) {
-            reader.releaseLock();
-          }
-        }
-      }
-      if (errorInRead !== "") {
-        setError(errorInRead);
-        reader.cancel();
-        setIdle(true);
-      }
-      return result;
-    });
-  }
-  return { response, send, stop, idle, error };
+var LLMServiceProvider = ({
+  children,
+  project_id,
+  customer
+}) => {
+  return /* @__PURE__ */ React.createElement(LLMService.Provider, { value: { project_id, customer } }, children);
 };
-var useLLM_default = useLLM;
 export {
-  LLMAsAService_default as LLMAsAService,
-  useLLM_default as useLLM
+  LLMService,
+  LLMServiceProvider
 };

package/index.ts

@@ -1,2 +1,2 @@
-export { default as useLLM } from './src/useLLM';
-export { default as LLMAsAService } from './src/LLMAsAService';
+export * from './src/useLLM';
+export * from './src/LLMAsAService';

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "llmasaservice-client",
   "license": "MIT",
-  "version": "0.0.1",
+  "version": "0.0.2",
   "main": "dist/index.js",
   "module": "dist/index.mjs",
   "types": "dist/index.d.ts",
@@ -22,5 +22,13 @@
     "typescript": "^5.5.3"
   },
   "description": "HOC and hook to use the LLMAsAService.io LLM load balancer and firewall",
-  "author": "CASEY, Inc. <support@heycasey.io>"
+  "author": "CASEY, Inc. <help@heycasey.io>",
+  "keywords": [
+    "react",
+    "llmasaservice",
+    "llm",
+    "openAI",
+    "chat"
+  ],
+  "homepage": "https://llmasaservice.io"
 }

package/src/LLMAsAService.tsx

@@ -1,6 +1,6 @@
 import React, { createContext, ReactNode } from "react";
 
-export type LLMAsAServiceCustomer = {
+type LLMAsAServiceCustomer = {
   customer_id: string;
   customer_name?: string;
   customer_user_id?: string;
@@ -9,7 +9,7 @@ export type LLMAsAServiceCustomer = {
 
 export interface LLMServiceType {
   project_id: string | undefined;
-  customer: LLMAsAServiceCustomer;
+  customer?: LLMAsAServiceCustomer;
 }
 
 export const LLMService = createContext<LLMServiceType | undefined>(undefined);
@@ -17,7 +17,7 @@ export const LLMService = createContext<LLMServiceType | undefined>(undefined);
 interface UserProviderProps {
   children: ReactNode;
   project_id: string | undefined;
-  customer: LLMAsAServiceCustomer;
+  customer?: LLMAsAServiceCustomer;
 }
 
 export const LLMServiceProvider: React.FC<UserProviderProps> = ({
@@ -31,5 +31,3 @@ export const LLMServiceProvider: React.FC<UserProviderProps> = ({
     </LLMService.Provider>
   );
 };
-
-export default LLMService;

package/src/useLLM.ts

@@ -6,7 +6,11 @@ const useLLM = (options?: LLMServiceType) => {
   const [idle, setIdle] = useState<boolean>(true);
   const [error, setError] = useState<string>("");
 
-  const context = useContext(LLMService) ?? options;
+  let context = useContext(LLMService);
+  if (!context) {
+    context = options;
+  }
+
   if (!context) {
     throw new Error(
       "useLLM must be used within a LLMServiceProvider or constructed with options in your useLLM() call."
@@ -51,7 +55,7 @@ const useLLM = (options?: LLMServiceType) => {
       serviceId: service,
       prompt: prompt,
       messages: messages,
-      customer: context?.customer ?? {},
+      customer: context?.customer ?? {}, // if no customer, use the projectId as the customer_id
     });
 
     // trying to get cloudfront oac going. posts need to be signed, but when i add this the call fails...