@aj-archipelago/cortex 0.0.11 → 1.0.1

package/LICENSE

@@ -1,10 +1,6 @@
 MIT License
 
-<<<<<<< HEAD
-Copyright (c) 2022 Al Jazeera Media Network
-=======
-Copyright (c) 2023 aj-archipelago
->>>>>>> b78a6667176ebfd09d2c7ed1549d8dc18691c344
+Copyright (c) 2023 Al Jazeera Media Network
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

package/README.md

@@ -5,7 +5,7 @@ Modern AI models are transformational, but a number of complexities emerge when
 ## Features
 
 * Simple architecture to build custom functional endpoints (called `pathways`), that implement common NL AI tasks. Default pathways include chat, summarization, translation, paraphrasing, completion, spelling and grammar correction, entity extraction, sentiment analysis, and bias analysis.
-* Allows for building multi-model, multi-vendor, and model-agnostic pathways (choose the right model or combination of models for the job, implement redundancy) with built-in support for OpenAI GPT-3, GPT-3.5 (chatGPT), and GPT-4 models - both from OpenAI directly and through Azure OpenAI, OpenAI Whisper, Azure Translator, and more.
+* Allows for building multi-model, multi-tool, multi-vendor, and model-agnostic pathways (choose the right model or combination of models and tools for the job, implement redundancy) with built-in support for OpenAI GPT-3, GPT-3.5 (chatGPT), and GPT-4 models - both from OpenAI directly and through Azure OpenAI, OpenAI Whisper, Azure Translator, LangChain.js and more.
 * Easy, templatized prompt definition with flexible support for most prompt engineering techniques and strategies ranging from simple single prompts to complex custom prompt chains with context continuity.
 * Built in support for long-running, asynchronous operations with progress updates or streaming responses
 * Integrated context persistence: have your pathways "remember" whatever you want and use it on the next request to the model
@@ -15,7 +15,7 @@ Modern AI models are transformational, but a number of complexities emerge when
 * Caching of repeated queries to provide instant results and avoid excess requests to the underlying model in repetitive use cases (chat bots, unit tests, etc.)
 
 ## Installation
-In order to use Cortex, you must first have a working Node.js environment. The version of Node.js should be at least 14 or higher. After verifying that you have the correct version of Node.js installed, you can get the simplest form up and running with a couple of commands.
+In order to use Cortex, you must first have a working Node.js environment. The version of Node.js should be 18 or higher (lower versions supported with some reduction in features). After verifying that you have the correct version of Node.js installed, you can get the simplest form up and running with a couple of commands.
 ## Quick Start
 ```sh
 git clone git@github.com:aj-archipelago/cortex.git
@@ -55,18 +55,20 @@ apolloClient.query({
 ## Cortex Pathways: Supercharged Prompts
 Pathways are a core concept in Cortex. Each pathway is a single JavaScript file that encapsulates the data and logic needed to define a functional API endpoint. When the client makes a request via the API, one or more pathways are executed and the result is sent back to the client. Pathways can be very simple:
 ```js
-module.exports = {
-    prompt: `{{text}}\n\nRewrite the above using British English spelling:`
+export default {
+  prompt: `{{text}}\n\nRewrite the above using British English spelling:`
 }
 ```
 The real power of Cortex starts to show as the pathways get more complex. This pathway, for example, uses a three-part sequential prompt to ensure that specific people and place names are correctly translated:
 ```js
-prompt:
-    [
-        `{{{text}}}\nCopy the names of all people and places exactly from this document in the language above:\n`,
-        `Original Language:\n{{{previousResult}}}\n\n{{to}}:\n`,
-        `Entities in the document:\n\n{{{previousResult}}}\n\nDocument:\n{{{text}}}\nRewrite the document in {{to}}. If the document is already in {{to}}, copy it exactly below:\n`
-    ]
+export default {
+  prompt:
+      [
+          `{{{text}}}\nCopy the names of all people and places exactly from this document in the language above:\n`,
+          `Original Language:\n{{{previousResult}}}\n\n{{to}}:\n`,
+          `Entities in the document:\n\n{{{previousResult}}}\n\nDocument:\n{{{text}}}\nRewrite the document in {{to}}. If the document is already in {{to}}, copy it exactly below:\n`
+      ]
+}
 ```
 Cortex pathway prompt enhancements include:
 * **Templatized prompt definition**: Pathways allow for easy and flexible prompt definition using Handlebars templating. This makes it simple to create and modify prompts using variables and context from the application as well as extensible internal functions provided by Cortex.
@@ -109,7 +111,7 @@ If you look closely at the examples above, you'll notice embedded parameters lik
 ### Parameters
 Pathways support an arbitrary number of input parameters.  These are defined in the pathway like this:
 ```js
-module.exports = {
+export default {
     prompt:
         [
             `{{{chatContext}}}\n\n{{{text}}}\n\nGiven the information above, create a short summary of the conversation to date making sure to include all of the personal details about the user that you encounter:\n\n`,            
@@ -161,7 +163,7 @@ A core function of Cortex is dealing with token limited interfaces. To this end,
 
 Cortex provides built in functions to turn loosely formatted text output from the model API calls into structured objects for return to the application. Specifically, Cortex provides parsers for numbered lists of strings and numbered lists of objects. These are used in pathways like this:
 ```js
-module.exports = {
+export default {
     temperature: 0,
     prompt: `{{text}}\n\nList the top {{count}} entities and their definitions for the above in the format {{format}}:`,
     format: `(name: definition)`,
@@ -179,47 +181,131 @@ The resolver property defines the function that processes the input and returns
 
 The core pathway `summary.js` below is implemented using custom pathway logic and a custom resolver to effectively target a specific summary length:
 ```js
-const { semanticTruncate } = require('../graphql/chunker');
-const { PathwayResolver } = require('../graphql/pathwayResolver');
-module.exports = {
-    prompt: `{{{text}}}\n\nWrite a summary of the above text:\n\n`,
+// summary.js
+// Text summarization module with custom resolver
+// This module exports a prompt that takes an input text and generates a summary using a custom resolver.
+
+// Import required modules
+import { semanticTruncate } from '../graphql/chunker.js';
+import { PathwayResolver } from '../graphql/pathwayResolver.js';
+
+export default {
+    // The main prompt function that takes the input text and asks to generate a summary.
+    prompt: `{{{text}}}\n\nWrite a summary of the above text. If the text is in a language other than english, make sure the summary is written in the same language:\n\n`,
+
+    // Define input parameters for the prompt, such as the target length of the summary.
     inputParameters: {
-        targetLength: 500,
+        targetLength: 0,
     },
+
+    // Custom resolver to generate summaries by reprompting if they are too long or too short.
     resolver: async (parent, args, contextValue, info) => {
         const { config, pathway, requestState } = contextValue;
         const originalTargetLength = args.targetLength;
-        const errorMargin = 0.2;
+
+        // If targetLength is not provided, execute the prompt once and return the result.
+        if (originalTargetLength === 0) {
+            let pathwayResolver = new PathwayResolver({ config, pathway, args, requestState });
+            return await pathwayResolver.resolve(args);
+        }
+
+        const errorMargin = 0.1;
         const lowTargetLength = originalTargetLength * (1 - errorMargin);
         const targetWords = Math.round(originalTargetLength / 6.6);
-        // if the text is shorter than the summary length, just return the text
+
+        // If the text is shorter than the summary length, just return the text.
         if (args.text.length <= originalTargetLength) {
             return args.text;
         }
+
         const MAX_ITERATIONS = 5;
         let summary = '';
-        let bestSummary = '';
-        let pathwayResolver = new PathwayResolver({ config, pathway, requestState });
-        // modify the prompt to be words-based instead of characters-based
-        pathwayResolver.pathwayPrompt = `{{{text}}}\n\nWrite a summary of the above text in exactly ${targetWords} words:\n\n`
+        let pathwayResolver = new PathwayResolver({ config, pathway, args, requestState });
+
+        // Modify the prompt to be words-based instead of characters-based.
+        pathwayResolver.pathwayPrompt = `Write a summary of all of the text below. If the text is in a language other than english, make sure the summary is written in the same language. Your summary should be ${targetWords} words in length.\n\nText:\n\n{{{text}}}\n\nSummary:\n\n`
+
         let i = 0;
-        // reprompt if summary is too long or too short
-        while (((summary.length > originalTargetLength) || (summary.length < lowTargetLength)) && i < MAX_ITERATIONS) {
+        // Make sure it's long enough to start
+        while ((summary.length < lowTargetLength) && i < MAX_ITERATIONS) {
             summary = await pathwayResolver.resolve(args);
             i++;
         }
-        // if the summary is still too long, truncate it
+
+        // If it's too long, it could be because the input text was chunked
+        // and now we have all the chunks together. We can summarize that
+        // to get a comprehensive summary.
+        if (summary.length > originalTargetLength) {
+            pathwayResolver.pathwayPrompt = `Write a summary of all of the text below. If the text is in a language other than english, make sure the summary is written in the same language. Your summary should be ${targetWords} words in length.\n\nText:\n\n${summary}\n\nSummary:\n\n`
+            summary = await pathwayResolver.resolve(args);
+            i++;
+
+            // Now make sure it's not too long
+            while ((summary.length > originalTargetLength) && i < MAX_ITERATIONS) {
+                pathwayResolver.pathwayPrompt = `${summary}\n\nIs that less than ${targetWords} words long? If not, try again using a length of no more than ${targetWords} words.\n\n`;
+                summary = await pathwayResolver.resolve(args);
+                i++;
+            }
+        }
+
+        // If the summary is still too long, truncate it.
         if (summary.length > originalTargetLength) {
             return semanticTruncate(summary, originalTargetLength);
         } else {
             return summary;
         }
     }
-}
+};
 ```
+### LangChain.js Support
+The ability to define a custom resolver function in Cortex pathways gives Cortex the flexibility to be able to cleanly incorporate alternate pipelines and technology stacks into the execution of a pathway.  LangChain JS (https://github.com/hwchase17/langchainjs) is a very popular and well supported mechanism for wiring together models, tools, and logic to achieve some amazing results.  We have developed specific functionality to support LangChain in the Cortex prompt execution framework and will continue to build features to fully integrate it with Cortex prompt execution contexts.
+
+Below is an example pathway integrating with one of the example agents from the LangChain docs.  You can see the seamless integration of Cortex's configuration and graphQL / REST interface logic.
+```js
+// lc_test.js
+// LangChain Cortex integration test
+
+// Import required modules
+import { OpenAI } from "langchain/llms";
+import { initializeAgentExecutor } from "langchain/agents";
+import { SerpAPI, Calculator } from "langchain/tools";
+
+export default {
+
+    // Implement custom logic and interaction with Cortex
+    // in custom resolver.
+    
+    resolver: async (parent, args, contextValue, info) => {
+
+        const { config } = contextValue;
+        const openAIApiKey = config.get('openaiApiKey');
+        const serpApiKey = config.get('serpApiKey');
+
+        const model = new OpenAI({ openAIApiKey: openAIApiKey, temperature: 0 });
+        const tools = [new SerpAPI( serpApiKey ), new Calculator()];
+
+        const executor = await initializeAgentExecutor(
+            tools,
+            model,
+            "zero-shot-react-description"
+            );
+
+        console.log(`====================`);
+        console.log("Loaded langchain agent.");
+        const input = args.text;
+        console.log(`Executing with input "${input}"...`);
+        const result = await executor.call({ input });
+        console.log(`Got output ${result.output}`);
+        console.log(`====================`);
+
+        return result?.output;
+    },
+};
+```
+
 ### Building and Loading Pathways
 
-Pathways are loaded from modules in the `pathways` directory. The pathways are built and loaded to the `config` object using the `buildPathways` function. The `buildPathways` function loads the base pathway, the core pathways, and any custom pathways. It then creates a new object that contains all the pathways and adds it to the pathways property of the config object. The order of loading means that custom pathways will always override any core pathways that Cortext provides. While pathways are designed to be self-contained, you can override some pathway properties - including whether they're even available at all - in the `pathways` section of the config file.
+Pathways are loaded from modules in the `pathways` directory. The pathways are built and loaded to the `config` object using the `buildPathways` function. The `buildPathways` function loads the base pathway, the core pathways, and any custom pathways. It then creates a new object that contains all the pathways and adds it to the pathways property of the config object. The order of loading means that custom pathways will always override any core pathways that Cortex provides. While pathways are designed to be self-contained, you can override some pathway properties - including whether they're even available at all - in the `pathways` section of the config file.
 
 ## Core (Default) Pathways
 
@@ -236,7 +322,84 @@ Below are the default pathways provided with Cortex. These can be used as is, ov
 - `translate`: Translates text from one language to another
 ## Extensibility
 
-Cortex is designed to be highly extensible. This allows you to customize the API to fit your needs. You can add new features, modify existing features, and even add integrations with other APIs and models.
+Cortex is designed to be highly extensible. This allows you to customize the API to fit your needs. You can add new features, modify existing features, and even add integrations with other APIs and models.  Here's an example of what an extended project might look like:
+
+### Cortex Internal Implementation
+
+- **config**
+  - default.json
+- package-lock.json
+- package.json
+- **pathways**
+  - chat_code.js
+  - chat_context.js
+  - chat_persist.js
+  - expand_story.js
+  - ...whole bunch of custom pathways
+  - translate_gpt4.js
+  - translate_turbo.js
+- start.js
+
+Where `default.json` holds all of your specific configuration:
+```js
+{
+    "defaultModelName": "oai-gpturbo",
+    "models": {
+        "oai-td3": {
+            "type": "OPENAI-COMPLETION",
+            "url": "https://api.openai.com/v1/completions",
+            "headers": {
+                "Authorization": "Bearer {{OPENAI_API_KEY}}",
+                "Content-Type": "application/json"
+            },
+            "params": {
+                "model": "text-davinci-003"
+            },
+            "requestsPerSecond": 10,
+            "maxTokenLength": 4096
+        },
+        "oai-gpturbo": {
+            "type": "OPENAI-CHAT",
+            "url": "https://api.openai.com/v1/chat/completions",
+            "headers": {
+                "Authorization": "Bearer {{OPENAI_API_KEY}}",
+                "Content-Type": "application/json"
+            },
+            "params": {
+                "model": "gpt-3.5-turbo"
+            },
+            "requestsPerSecond": 10,
+            "maxTokenLength": 8192
+        },
+        "oai-gpt4": {
+            "type": "OPENAI-CHAT",
+            "url": "https://api.openai.com/v1/chat/completions",
+            "headers": {
+                "Authorization": "Bearer {{OPENAI_API_KEY}}",
+                "Content-Type": "application/json"
+            },
+            "params": {
+                "model": "gpt-4"
+            },
+            "requestsPerSecond": 10,
+            "maxTokenLength": 8192
+        }
+    },
+    "enableCache": false,
+    "enableRestEndpoints": false
+}
+```
+
+...and `start.js` is really simple:
+```js
+import cortex from '@aj-archipelago/cortex';
+
+(async () => {
+  const { startServer } = await cortex();
+  startServer && startServer();
+})();
+```
+
 ## Configuration
 Configuration of Cortex is done via a [convict](https://github.com/mozilla/node-convict/tree/master) object called `config`. The `config` object is built by combining the default values and any values specified in a configuration file or environment variables. The environment variables take precedence over the values in the configuration file. Below are the configurable properties and their defaults:
 
@@ -272,7 +435,7 @@ If you encounter any issues while using Cortex, there are a few things you can d
 If you would like to contribute to Cortex, there are two ways to do so. You can submit issues to the Cortex GitHub repository or submit pull requests with your proposed changes.
 
 ## License
-Cortex is released under the MIT License. See [LICENSE](https://github.com/ALJAZEERAPLUS/cortex/blob/main/LICENSE) for more details.
+Cortex is released under the MIT License. See [LICENSE](https://github.com/aj-archipelago/cortex/blob/main/LICENSE) for more details.
 
 ## API Reference
 Detailed documentation on Cortex's API can be found in the /graphql endpoint of your project. Examples of queries and responses can also be found in the Cortex documentation, along with tips for getting the most out of Cortex.
@@ -280,5 +443,6 @@ Detailed documentation on Cortex's API can be found in the /graphql endpoint of
 ## Roadmap
 Cortex is a constantly evolving project, and the following features are coming soon:
 
+* Prompt execution context preservation between calls (to enable interactive, multi-call integrations with LangChain and other technologies)
 * Model-specific cache key optimizations to increase hit rate and reduce cache size
 * Structured analytics and reporting on AI API call frequency, cost, cache hit rate, etc.

package/SECURITY.md

@@ -0,0 +1,22 @@
+# Security Policy
+
+## Supported Versions
+
+We take the security of our project seriously. The table below shows the versions of Cortex currently being supported with security updates.
+
+| Version | Supported          |
+| ------- | ------------------ |
+| 1.x.x   | :white_check_mark: |
+
+## Reporting a Vulnerability
+
+If you have discovered a security vulnerability in Cortex, please follow these steps to report it:
+
+1. **Do not** create a public GitHub issue, as this might expose the vulnerability to others.
+2. Please follow the GitHub process for [Privately Reporting a Security Vulnerability](https://docs.github.com/en/code-security/security-advisories/guidance-on-reporting-and-writing/privately-reporting-a-security-vulnerability)
+
+## Disclosure Policy
+
+Cortex follows responsible disclosure practices. Once a vulnerability is confirmed and a fix is developed, we will release a security update and publicly disclose the vulnerability. We will credit the reporter of the vulnerability in the disclosure, unless the reporter wishes to remain anonymous.
+
+We appreciate your help in keeping Cortex secure and your responsible disclosure of any security vulnerabilities you discover.

package/graphql/pathwayResolver.js

@@ -151,7 +151,7 @@ class PathwayResolver {
         }
         const encoded = encode(text);
         if (!this.useInputChunking || encoded.length <= chunkTokenLength) { // no chunking, return as is
-            if (encoded.length >= chunkTokenLength) {
+            if (encoded.length > 0 && encoded.length >= chunkTokenLength) {
                 const warnText = `Truncating long input text. Text length: ${text.length}`;
                 this.warnings.push(warnText);
                 console.warn(warnText);
@@ -189,7 +189,7 @@ class PathwayResolver {
         // the token ratio is the ratio of the total prompt to the result text - both have to be included
         // in computing the max token length
         const promptRatio = this.pathwayPrompter.plugin.getPromptTokenRatio();
-        let chunkMaxTokenLength = promptRatio * this.pathwayPrompter.plugin.getModelMaxTokenLength() - maxPromptTokenLength;
+        let chunkMaxTokenLength = promptRatio * this.pathwayPrompter.plugin.getModelMaxTokenLength() - maxPromptTokenLength - 1;
         
         // if we have to deal with prompts that have both text input
         // and previous result, we need to split the maxChunkToken in half

package/graphql/plugins/localModelPlugin.js

@@ -0,0 +1,23 @@
+// localModelPlugin.js
+import ModelPlugin from './modelPlugin.js';
+import { execFileSync } from 'child_process';
+
+class LocalModelPlugin extends ModelPlugin {
+    constructor(config, pathway) {
+        super(config, pathway);
+    }
+
+    async execute(text, parameters, prompt, pathwayResolver) {
+        const { modelPromptText } = this.getCompiledPrompt(text, parameters, prompt);
+
+        try {
+            const result = execFileSync(executablePath, [text], { encoding: 'utf8' });
+            return result;
+        } catch (error) {
+            console.error('Error running local model:', error);
+            throw error;
+        }
+    }
+}
+
+export default LocalModelPlugin;

package/graphql/plugins/openAiChatPlugin.js

@@ -12,14 +12,14 @@ class OpenAIChatPlugin extends ModelPlugin {
         const { stream } = parameters;
     
         // Define the model's max token length
-        const modelMaxTokenLength = this.getModelMaxTokenLength() * this.getPromptTokenRatio();
+        const modelTargetTokenLength = this.getModelMaxTokenLength() * this.getPromptTokenRatio();
     
         let requestMessages = modelPromptMessages || [{ "role": "user", "content": modelPromptText }];
     
         // Check if the token length exceeds the model's max token length
-        if (tokenLength > modelMaxTokenLength) {
+        if (tokenLength > modelTargetTokenLength) {
             // Remove older messages until the token length is within the model's limit
-            requestMessages = this.truncateMessagesToTargetLength(requestMessages, modelMaxTokenLength);
+            requestMessages = this.truncateMessagesToTargetLength(requestMessages, modelTargetTokenLength);
         }
     
         const requestParameters = {

package/graphql/plugins/openAiCompletionPlugin.js

@@ -13,19 +13,22 @@ class OpenAICompletionPlugin extends ModelPlugin {
         let { modelPromptMessages, modelPromptText, tokenLength } = this.getCompiledPrompt(text, parameters, prompt);
         const { stream } = parameters;
         let modelPromptMessagesML = '';
-        const modelMaxTokenLength = this.getModelMaxTokenLength();
+        // Define the model's max token length
+        const modelTargetTokenLength = this.getModelMaxTokenLength() * this.getPromptTokenRatio();
         let requestParameters = {};
     
         if (modelPromptMessages) {
-            const requestMessages = this.truncateMessagesToTargetLength(modelPromptMessages, modelMaxTokenLength - 1);
+            const minMsg = [{ role: "system", content: "" }];
+            const addAssistantTokens = encode(this.messagesToChatML(minMsg, true).replace(this.messagesToChatML(minMsg, false), '')).length;
+            const requestMessages = this.truncateMessagesToTargetLength(modelPromptMessages, (modelTargetTokenLength - addAssistantTokens));
             modelPromptMessagesML = this.messagesToChatML(requestMessages);
             tokenLength = encode(modelPromptMessagesML).length;
         
-            if (tokenLength >= modelMaxTokenLength) {
-                throw new Error(`The maximum number of tokens for this model is ${modelMaxTokenLength}. Please reduce the number of messages in the prompt.`);
+            if (tokenLength > modelTargetTokenLength) {
+                throw new Error(`Input is too long at ${tokenLength} tokens (this target token length for this pathway is ${modelTargetTokenLength} tokens because the response is expected to take up the rest of the model's max tokens (${this.getModelMaxTokenLength()}). You must reduce the size of the prompt to continue.`);
             }
         
-            const max_tokens = modelMaxTokenLength - tokenLength - 1;
+            const max_tokens = this.getModelMaxTokenLength() - tokenLength;
         
             requestParameters = {
                 prompt: modelPromptMessagesML,
@@ -38,11 +41,11 @@ class OpenAICompletionPlugin extends ModelPlugin {
                 stream
             };
         } else {
-            if (tokenLength >= modelMaxTokenLength) {
-                throw new Error(`The maximum number of tokens for this model is ${modelMaxTokenLength}. Please reduce the length of the prompt.`);
+            if (tokenLength > modelTargetTokenLength) {
+                throw new Error(`Input is too long at ${tokenLength} tokens. The target token length for this pathway is ${modelTargetTokenLength} tokens because the response is expected to take up the rest of the ${this.getModelMaxTokenLength()} tokens that the model can handle. You must reduce the size of the prompt to continue.`);
             }
         
-            const max_tokens = modelMaxTokenLength - tokenLength - 1;
+            const max_tokens = this.getModelMaxTokenLength() - tokenLength;
         
             requestParameters = {
                 prompt: modelPromptText,

package/graphql/plugins/openAiWhisperPlugin.js

@@ -1,4 +1,4 @@
-// OpenAICompletionPlugin.js
+// openAiWhisperPlugin.js
 import ModelPlugin from './modelPlugin.js';
 
 import FormData from 'form-data';

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@aj-archipelago/cortex",
-  "version": "0.0.11",
+  "version": "1.0.1",
   "description": "Cortex is a GraphQL API for AI. It provides a simple, extensible interface for using AI services from OpenAI, Azure and others.",
   "repository": {
     "type": "git",