@aj-archipelago/cortex 1.0.1 → 1.0.3

package/.eslintignore

@@ -0,0 +1,30 @@
+# Ignore build artifacts
+/dist
+/build
+
+# Ignore node_modules
+/node_modules
+
+# Ignore log files
+*.log
+
+# Ignore any config files
+.env
+.env.*
+
+# Ignore coverage reports
+/coverage
+
+# Ignore documentation
+/docs
+
+# Ignore any generated or bundled files
+*.min.js
+*.bundle.js
+
+# Ignore any files generated by your IDE or text editor
+.idea/
+.vscode/
+*.sublime-*
+*.iml
+*.swp

package/.eslintrc

@@ -0,0 +1,31 @@
+{
+    "env": {
+      "browser": true,
+      "es2021": true,
+        "node": true
+    },
+    "extends": [
+      "eslint:recommended"
+    ],
+    "parserOptions": {
+      "ecmaVersion": "latest",
+      "sourceType": "module"
+    },
+    "plugins": [
+      "import"
+    ],
+    "rules": {
+        "import/no-unresolved": "error",
+        "import/no-extraneous-dependencies": ["error", {"devDependencies": true}],
+        "no-unused-vars": ["error", { "argsIgnorePattern": "^_" }]
+    },
+    "settings": {
+        "import/resolver": {
+          "node": {
+            "extensions": [".js"],
+            "moduleDirectory": ["node_modules", "src"]
+          }
+        },
+        "import/core-modules": ["ava"]
+    }
+  }

package/README.md

@@ -2,6 +2,18 @@
 Cortex simplifies and accelerates the process of creating applications that harness the power of modern AI models like chatGPT and GPT-4 by providing a structured interface (GraphQL or REST) to a powerful prompt execution environment. This enables complex augmented prompting and abstracts away most of the complexity of managing model connections like chunking input, rate limiting, formatting output, caching, and handling errors.
 ## Why build Cortex?
 Modern AI models are transformational, but a number of complexities emerge when developers start using them to deliver application-ready functions. Most models require precisely formatted, carefully engineered and sequenced prompts to produce consistent results, and the responses are typically largely unstructured text without validation or formatting. Additionally, these models are evolving rapidly, are typically costly and slow to query and implement hard request size and rate restrictions that need to be carefully navigated for optimum throughput. Cortex offers a solution to these problems and provides a simple and extensible package for interacting with NL AI models.
+
+## Okay, but what can I really do with this thing?
+Just about anything! It's kind of an LLM swiss army knife.  Here are some ideas:
+* Create custom chat agents with memory and personalization and then expose them through a bunch of different UIs (custom chat portals, Slack, Microsoft Teams, etc. - anything that can be extended and speak to a REST or GraphQL endpoint)
+* Spin up LLM powered automatons with their prompting logic and AI API handling logic all centrally encapsulated.
+* Make LLM chains and agents from LangChain.js available via scalable REST or GraphQL endpoints.
+* Put a REST or GraphQL front end on your locally-run models (e.g. llama.cpp) and use them in concert with other tools.
+* Create modular custom coding assistants (code generation, code reviews, test writing, AI pair programming) and easily integrate them with your existing editing tools.
+* Create powerful AI editing tools (copy editing, paraphrasing, summarization, etc.) for your company and then integrate them with your existing workflow tools without having to build all the LLM-handling logic into those tools.
+* Create cached endpoints for functions with repeated calls so the results return instantly and you don't run up LLM token charges.
+* Route all of your company's LLM access through a single API layer to optimize and monitor usage and centrally control rate limiting and which models are being used.
+
 ## Features
 
 * Simple architecture to build custom functional endpoints (called `pathways`), that implement common NL AI tasks. Default pathways include chat, summarization, translation, paraphrasing, completion, spelling and grammar correction, entity extraction, sentiment analysis, and bias analysis.
@@ -24,7 +36,7 @@ npm install
 export OPENAI_API_KEY=<your key>
 npm start
 ```
-Yup, that's it, at least in the simplest possible case. That will get you access to all of the built in pathways.
+Yup, that's it, at least in the simplest possible case. That will get you access to all of the built in pathways.  If you prefer to use npm instead instead of cloning, we have an npm package too: [@aj-archipelago/cortex](https://www.npmjs.com/package/@aj-archipelago/cortex)
 ## Connecting Applications to Cortex
 Cortex speaks GraphQL and by default it enables the GraphQL playground. If you're just using default options, that's at [http://localhost:4000/graphql](http://localhost:4000/graphql). From there you can begin making requests and test out the pathways (listed under Query) to your heart's content. If GraphQL isn't your thing or if you have a client that would rather have REST that's fine - Cortex speaks REST as well.
 

package/config/default.example.json

@@ -0,0 +1,70 @@
+{
+    "defaultModelName": "oai-td3",
+    "models": {
+        "azure-translate": {
+            "type": "AZURE-TRANSLATE",
+            "url": "https://api.cognitive.microsofttranslator.com/translate?api-version=3.0",
+            "headers": {
+                "Ocp-Apim-Subscription-Key": "{{ARCHIPELAGO_TRANSLATE_KEY}}",
+                "Ocp-Apim-Subscription-Region": "eastus",
+                "Content-Type": "application/json"
+            },
+            "requestsPerSecond": 10,
+            "maxTokenLength": 2000
+        },
+        "oai-td3": {
+            "type": "OPENAI-COMPLETION",
+            "url": "https://api.openai.com/v1/completions",
+            "headers": {
+                "Authorization": "Bearer {{OPENAI_API_KEY}}",
+                "Content-Type": "application/json"
+            },
+            "params": {
+                "model": "text-davinci-003"
+            },
+            "requestsPerSecond": 10,
+            "maxTokenLength": 4096
+        },
+        "oai-gpturbo": {
+            "type": "OPENAI-CHAT",
+            "url": "https://api.openai.com/v1/chat/completions",
+            "headers": {
+                "Authorization": "Bearer {{OPENAI_API_KEY}}",
+                "Content-Type": "application/json"
+            },
+            "params": {
+                "model": "gpt-3.5-turbo"
+            },
+            "requestsPerSecond": 10,
+            "maxTokenLength": 8192
+        },
+        "oai-gpt4": {
+            "type": "OPENAI-CHAT",
+            "url": "https://api.openai.com/v1/chat/completions",
+            "headers": {
+                "Authorization": "Bearer {{OPENAI_API_KEY}}",
+                "Content-Type": "application/json"
+            },
+            "params": {
+                "model": "gpt-4"
+            },
+            "requestsPerSecond": 10,
+            "maxTokenLength": 8192
+        },
+        "local-llama13B": {
+            "type": "LOCAL-CPP-MODEL",
+            "executablePath": "../llm/llama.cpp/main",
+            "args": [
+                "-m", "../llm/llama.cpp/models/13B/ggml-model-q4_0.bin",
+                "--repeat_penalty", "1.0",
+                "--keep", "0",
+                "-t", "8",
+                "--mlock"
+            ],
+            "requestsPerSecond": 10,
+            "maxTokenLength": 1024
+        }
+    },
+    "enableCache": false,
+    "enableRestEndpoints": false
+}

package/config.js

@@ -1,7 +1,7 @@
 import path from 'path';
 const __dirname = path.dirname(new URL(import.meta.url).pathname);
 import convict from 'convict';
-import handlebars from 'handlebars';
+import HandleBars from './lib/handleBars.js';
 import fs from 'fs';
 
 // Schema for config
@@ -110,11 +110,10 @@ var config = convict({
         default: null,
         env: 'CORTEX_CONFIG_FILE'
     },
-    serpApiKey: {
+    whisperMediaApiUrl: {
         format: String,
-        default: null,
-        env: 'SERPAPI_API_KEY',
-        sensitive: true
+        default: 'null',
+        env: 'WHISPER_MEDIA_API_URL'
     },
 });
 
@@ -173,7 +172,7 @@ const buildModels = (config) => {
 
     for (const [key, model] of Object.entries(models)) {        
         // Compile handlebars templates for models
-        models[key] = JSON.parse(handlebars.compile(JSON.stringify(model))({ ...config.getEnv(), ...config.getProperties() }))
+        models[key] = JSON.parse(HandleBars.compile(JSON.stringify(model))({ ...config.getEnv(), ...config.getProperties() }))
     }
 
     // Add constructed models to config

package/graphql/chunker.js

@@ -43,7 +43,7 @@ const getSemanticChunks = (text, chunkSize) => {
   };
 
   const breakByParagraphs = (str) => breakByRegex(str, /[\r\n]+/, true);
-  const breakBySentences = (str) => breakByRegex(str, /(?<=[.。؟！\?!\n])\s+/, true);
+  const breakBySentences = (str) => breakByRegex(str, /(?<=[.。؟！?!\n])\s+/, true);
   const breakByWords = (str) => breakByRegex(str, /(\s,;:.+)/);
 
   const createChunks = (tokens) => {

package/graphql/graphql.js

@@ -48,7 +48,7 @@ const buildRestEndpoints = (pathways, app, server, config) => {
 
     app.post(`/rest/${name}`, async (req, res) => {
       const variables = fieldVariableDefs.reduce((acc, variableDef) => {
-        if (req.body.hasOwnProperty(variableDef.name)) {
+        if (Object.prototype.hasOwnProperty.call(req.body, variableDef.name)) {
           acc[variableDef.name] = req.body[variableDef.name];
         }
         return acc;

package/graphql/parser.js

@@ -6,6 +6,7 @@ const regexParser = (text, regex) => {
 // parse numbered list text format into list
 // this supports most common numbered list returns like "1.", "1)", "1-"
 const parseNumberedList = (str) => {
+    // eslint-disable-next-line no-useless-escape
     return regexParser(str, /^\s*[\[\{\(]*\d+[\s.=\-:,;\]\)\}]/gm);
 }
 
@@ -31,8 +32,14 @@ const parseNumberedObjectList = (text, format) => {
     return result;
 }
 
+// parse a comma-separated list text format into list
+const parseCommaSeparatedList = (str) => {
+    return str.split(',').map(s => s.trim()).filter(s => s.length);
+}
+
 export {
     regexParser,
     parseNumberedList,
     parseNumberedObjectList,
+    parseCommaSeparatedList,
 };

package/graphql/pathwayPrompter.js

@@ -1,23 +1,9 @@
 // PathwayPrompter.js
-import OpenAIChatPlugin from './plugins/openAIChatPlugin.js';
-import OpenAICompletionPlugin from './plugins/openAICompletionPlugin.js';
+import OpenAIChatPlugin from './plugins/openAiChatPlugin.js';
+import OpenAICompletionPlugin from './plugins/openAiCompletionPlugin.js';
 import AzureTranslatePlugin from './plugins/azureTranslatePlugin.js';
 import OpenAIWhisperPlugin from './plugins/openAiWhisperPlugin.js';
-import handlebars from 'handlebars';
-
-// register functions that can be called directly in the prompt markdown
-handlebars.registerHelper('stripHTML', function (value) {
-    return value.replace(/<[^>]*>/g, '');
-});
-
-handlebars.registerHelper('now', function () {
-    return new Date().toISOString();
-});
-
-handlebars.registerHelper('toJSON', function (object) {
-    return JSON.stringify(object);
-});
-
+import LocalModelPlugin from './plugins/localModelPlugin.js';
 
 class PathwayPrompter {
     constructor({ config, pathway }) {
@@ -26,7 +12,7 @@ class PathwayPrompter {
         const model = config.get('models')[modelName];
 
         if (!model) {
-            throw new handlebars.Exception(`Model ${modelName} not found in config`);
+            throw new Error(`Model ${modelName} not found in config`);
         }
 
         let plugin;
@@ -44,8 +30,11 @@ class PathwayPrompter {
             case 'OPENAI_WHISPER':
                 plugin = new OpenAIWhisperPlugin(config, pathway);
                 break;
+            case 'LOCAL-CPP-MODEL':
+                plugin = new LocalModelPlugin(config, pathway);
+                break;
             default:
-                throw new handlebars.Exception(`Unsupported model type: ${model.type}`);
+                throw new Error(`Unsupported model type: ${model.type}`);
         }
 
         this.plugin = plugin;

package/graphql/pathwayResolver.js

@@ -1,4 +1,5 @@
 import { PathwayPrompter } from './pathwayPrompter.js';
+// eslint-disable-next-line import/no-extraneous-dependencies
 import { v4 as uuidv4 } from 'uuid';
 import pubsub from './pubsub.js';
 import { encode } from 'gpt-3-encoder';
@@ -7,13 +8,7 @@ import { PathwayResponseParser } from './pathwayResponseParser.js';
 import { Prompt } from './prompt.js';
 import { getv, setv } from '../lib/keyValueStorageClient.js';
 import { requestState } from './requestState.js';
-
-const MAX_PREVIOUS_RESULT_TOKEN_LENGTH = 1000;
-
-const callPathway = async (config, pathwayName, args, requestState, { text, ...parameters }) => {
-    const pathwayResolver = new PathwayResolver({ config, pathway: config.get(`pathways.${pathwayName}`), args, requestState });
-    return await pathwayResolver.resolve({ text, ...parameters });
-}
+import { callPathway } from '../lib/pathwayTools.js';
 
 class PathwayResolver {
     constructor({ config, pathway, args }) {
@@ -141,6 +136,12 @@ class PathwayResolver {
         return this.responseParser.parse(data);
     }
 
+    // Add a warning and log it
+    logWarning(warning) {
+        this.warnings.push(warning);
+        console.warn(warning);
+    }
+
     // Here we choose how to handle long input - either summarize or chunk
     processInputText(text) {
         let chunkTokenLength = 0;
@@ -153,8 +154,7 @@ class PathwayResolver {
         if (!this.useInputChunking || encoded.length <= chunkTokenLength) { // no chunking, return as is
             if (encoded.length > 0 && encoded.length >= chunkTokenLength) {
                 const warnText = `Truncating long input text. Text length: ${text.length}`;
-                this.warnings.push(warnText);
-                console.warn(warnText);
+                this.logWarning(warnText);
                 text = this.truncate(text, chunkTokenLength);
             }
             return [text];
@@ -173,7 +173,7 @@ class PathwayResolver {
 
     async summarizeIfEnabled({ text, ...parameters }) {
         if (this.pathway.useInputSummarization) {
-            return await callPathway(this.config, 'summary', this.args, requestState, { text, targetLength: 1000, ...parameters });
+            return await callPathway(this.config, 'summary', { ...this.args, ...parameters, targetLength: 0});
         }
         return text;
     }

package/graphql/pathwayResponseParser.js

@@ -1,20 +1,29 @@
-import { parseNumberedList, parseNumberedObjectList } from './parser.js';
+import { parseNumberedList, parseNumberedObjectList, parseCommaSeparatedList } from './parser.js';
 
 class PathwayResponseParser {
     constructor(pathway) {
         this.pathway = pathway;
     }
 
+    isCommaSeparatedList(data) {
+        const commaSeparatedPattern = /^([^,\n]+,)+[^,\n]+$/;
+        return commaSeparatedPattern.test(data.trim());
+    }
+
     parse(data) {
         if (this.pathway.parser) {
             return this.pathway.parser(data);
         }
 
         if (this.pathway.list) {
-            if (this.pathway.format) {
-                return parseNumberedObjectList(data, this.pathway.format);
+            if (this.isCommaSeparatedList(data)) {
+                return parseCommaSeparatedList(data);
+            } else {
+                if (this.pathway.format) {
+                    return parseNumberedObjectList(data, this.pathway.format);
+                }
+                return parseNumberedList(data);
             }
-            return parseNumberedList(data)
         }
 
         return data;

package/graphql/plugins/localModelPlugin.js

@@ -1,20 +1,69 @@
 // localModelPlugin.js
 import ModelPlugin from './modelPlugin.js';
 import { execFileSync } from 'child_process';
+import { encode } from 'gpt-3-encoder';
 
 class LocalModelPlugin extends ModelPlugin {
     constructor(config, pathway) {
         super(config, pathway);
     }
 
-    async execute(text, parameters, prompt, pathwayResolver) {
-        const { modelPromptText } = this.getCompiledPrompt(text, parameters, prompt);
+    // if the input starts with a chatML response, just return that
+    filterFirstResponse(inputString) {
+        const regex = /^(.*?)(?=\n<\|im_end\|>|$)/;
+        const match = inputString.match(regex);
+
+        if (match) {
+            const firstAssistantResponse = match[1];
+            return firstAssistantResponse;
+        } else {
+            return inputString;
+        }
+    }
+
+    getRequestParameters(text, parameters, prompt) {
+        let { modelPromptMessages, modelPromptText, tokenLength } = this.getCompiledPrompt(text, parameters, prompt);
+        const modelTargetTokenLength = this.getModelMaxTokenLength() * this.getPromptTokenRatio();
+        
+        if (modelPromptMessages) {
+            const minMsg = [{ role: "system", content: "" }];
+            const addAssistantTokens = encode(this.messagesToChatML(minMsg, true).replace(this.messagesToChatML(minMsg, false), '')).length;
+            const requestMessages = this.truncateMessagesToTargetLength(modelPromptMessages, (modelTargetTokenLength - addAssistantTokens));
+            modelPromptText = this.messagesToChatML(requestMessages);
+            tokenLength = encode(modelPromptText).length;
+        }
+
+        if (tokenLength > modelTargetTokenLength) {
+            throw new Error(`Input is too long at ${tokenLength} tokens. The target token length for this pathway is ${modelTargetTokenLength} tokens because the response is expected to take up the rest of the ${this.getModelMaxTokenLength()} tokens that the model can handle. You must reduce the size of the prompt to continue.`);
+        }
+
+        const max_tokens = this.getModelMaxTokenLength() - tokenLength;
+
+        return {
+            prompt: modelPromptText,
+            max_tokens: max_tokens,
+            temperature: this.temperature ?? 0.7,
+        };
+    }
+
+    async execute(text, parameters, prompt, _pathwayResolver) {
+        const requestParameters = this.getRequestParameters(text, parameters, prompt);
+        const { executablePath, args } = this.model;
+        args.push("--prompt", requestParameters.prompt);
+        //args.push("--max-tokens", requestParameters.max_tokens);
+        //args.push("--temperature", requestParameters.temperature);
 
         try {
-            const result = execFileSync(executablePath, [text], { encoding: 'utf8' });
-            return result;
+            console.log(`\x1b[36mRunning local model:\x1b[0m`, executablePath, args);
+            const result = execFileSync(executablePath, args, { encoding: 'utf8' });
+            // Remove only the first occurrence of requestParameters.prompt from the result
+            // Could have used regex here but then would need to escape the prompt
+            const parts = result.split(requestParameters.prompt, 2);
+            const modifiedResult = parts[0] + parts[1];
+            console.log(`\x1b[36mResult:\x1b[0m`, modifiedResult);
+            return this.filterFirstResponse(modifiedResult);
         } catch (error) {
-            console.error('Error running local model:', error);
+            console.error(`\x1b[31mError running local model:\x1b[0m`, error);
             throw error;
         }
     }

package/graphql/plugins/modelPlugin.js

@@ -1,5 +1,5 @@
 // ModelPlugin.js
-import handlebars from 'handlebars';
+import HandleBars from '../../lib/handleBars.js';
 
 import { request } from '../../lib/request.js';
 import { encode } from 'gpt-3-encoder';
@@ -40,7 +40,7 @@ class ModelPlugin {
         this.shouldCache = config.get('enableCache') && (pathway.enableCache || pathway.temperature == 0);
     }
 
-    truncateMessagesToTargetLength = (messages, targetTokenLength) => {
+    truncateMessagesToTargetLength(messages, targetTokenLength) {
         // Calculate the token length of each message
         const tokenLengths = messages.map((message) => ({
             message,
@@ -58,7 +58,7 @@ class ModelPlugin {
     
         // Remove and/or truncate messages until the target token length is reached
         let index = 0;
-        while (totalTokenLength > targetTokenLength) {
+        while ((totalTokenLength > targetTokenLength) && (index < tokenLengths.length)) {
             const message = tokenLengths[index].message;
 
             // Skip system messages
@@ -79,25 +79,34 @@ class ModelPlugin {
                 const otherMessageTokens = totalTokenLength - currentTokenLength;
                 const tokensToKeep = targetTokenLength - (otherMessageTokens + emptyContentLength);
 
-                const truncatedContent = getFirstNToken(message.content, tokensToKeep);
-                const truncatedMessage = { ...message, content: truncatedContent };
-            
-                tokenLengths[index] = {
-                    message: truncatedMessage,
-                    tokenLength: encode(this.messagesToChatML([ truncatedMessage ], false)).length
-                }
+                if (tokensToKeep <= 0) {
+                    // If the message needs to be empty to make the target, remove it entirely
+                    totalTokenLength -= currentTokenLength;
+                    tokenLengths.splice(index, 1);
+                } else {
+                    // Otherwise, update the message and token length
+                    const truncatedContent = getFirstNToken(message.content, tokensToKeep);
+                    const truncatedMessage = { ...message, content: truncatedContent };
+
+                    tokenLengths[index] = {
+                        message: truncatedMessage,
+                        tokenLength: encode(this.messagesToChatML([ truncatedMessage ], false)).length
+                    }
 
-                // calculate the length again to keep us honest
-                totalTokenLength = tokenLengths.reduce(
-                    (sum, { tokenLength }) => sum + tokenLength,
-                    0
-                );
+                    // calculate the length again to keep us honest
+                    totalTokenLength = tokenLengths.reduce(
+                        (sum, { tokenLength }) => sum + tokenLength,
+                        0
+                    );
+
+                    index++;
+                }
             }
         }
     
         // Return the modified messages array
         return tokenLengths.map(({ message }) => message);
-    };
+    }
     
     //convert a messages array to a simple chatML format
     messagesToChatML(messages, addAssistant = true) {
@@ -118,7 +127,7 @@ class ModelPlugin {
     getCompiledPrompt(text, parameters, prompt) {
         const combinedParameters = { ...this.promptParameters, ...parameters };
         const modelPrompt = this.getModelPrompt(prompt, parameters);
-        const modelPromptText = modelPrompt.prompt ? handlebars.compile(modelPrompt.prompt)({ ...combinedParameters, text }) : '';
+        const modelPromptText = modelPrompt.prompt ? HandleBars.compile(modelPrompt.prompt)({ ...combinedParameters, text }) : '';
         const modelPromptMessages = this.getModelPromptMessages(modelPrompt, combinedParameters, text);
         const modelPromptMessagesML = this.messagesToChatML(modelPromptMessages);
 
@@ -135,7 +144,7 @@ class ModelPlugin {
 
     getPromptTokenRatio() {
         // TODO: Is this the right order of precedence? inputParameters should maybe be second?
-        return this.promptParameters.inputParameters.tokenRatio ?? this.promptParameters.tokenRatio ?? DEFAULT_PROMPT_TOKEN_RATIO;
+        return this.promptParameters.inputParameters?.tokenRatio ?? this.promptParameters.tokenRatio ?? DEFAULT_PROMPT_TOKEN_RATIO;
     }
 
 
@@ -155,7 +164,7 @@ class ModelPlugin {
         // First run handlebars compile on the pathway messages
         const compiledMessages = modelPrompt.messages.map((message) => {
             if (message.content) {
-                const compileText = handlebars.compile(message.content);
+                const compileText = HandleBars.compile(message.content);
                 return {
                     role: message.role,
                     content: compileText({ ...combinedParameters, text }),
@@ -184,7 +193,7 @@ class ModelPlugin {
     }
 
     requestUrl() {
-        const generateUrl = handlebars.compile(this.model.url);
+        const generateUrl = HandleBars.compile(this.model.url);
         return generateUrl({ ...this.model, ...this.environmentVariables, ...this.config });
     }
 

package/graphql/plugins/openAiCompletionPlugin.js

@@ -1,15 +1,27 @@
 // OpenAICompletionPlugin.js
+
 import ModelPlugin from './modelPlugin.js';
 
 import { encode } from 'gpt-3-encoder';
 
+// Helper function to truncate the prompt if it is too long
+const truncatePromptIfNecessary = (text, textTokenCount, modelMaxTokenCount, targetTextTokenCount, pathwayResolver) => {
+    const maxAllowedTextTokenCount = textTokenCount + ((modelMaxTokenCount - targetTextTokenCount) * 0.5);
+
+    if (textTokenCount > maxAllowedTextTokenCount) {
+        pathwayResolver.logWarning(`Prompt is too long at ${textTokenCount} tokens (this target token length for this pathway is ${targetTextTokenCount} tokens because the response is expected to take up the rest of the model's max tokens (${modelMaxTokenCount}). Prompt will be truncated.`);
+        return pathwayResolver.truncate(text, maxAllowedTextTokenCount);
+    }
+    return text;
+}
+
 class OpenAICompletionPlugin extends ModelPlugin {
     constructor(config, pathway) {
         super(config, pathway);
     }
 
     // Set up parameters specific to the OpenAI Completion API
-    getRequestParameters(text, parameters, prompt) {
+    getRequestParameters(text, parameters, prompt, pathwayResolver) {
         let { modelPromptMessages, modelPromptText, tokenLength } = this.getCompiledPrompt(text, parameters, prompt);
         const { stream } = parameters;
         let modelPromptMessagesML = '';
@@ -23,12 +35,14 @@ class OpenAICompletionPlugin extends ModelPlugin {
             const requestMessages = this.truncateMessagesToTargetLength(modelPromptMessages, (modelTargetTokenLength - addAssistantTokens));
             modelPromptMessagesML = this.messagesToChatML(requestMessages);
             tokenLength = encode(modelPromptMessagesML).length;
-        
-            if (tokenLength > modelTargetTokenLength) {
-                throw new Error(`Input is too long at ${tokenLength} tokens (this target token length for this pathway is ${modelTargetTokenLength} tokens because the response is expected to take up the rest of the model's max tokens (${this.getModelMaxTokenLength()}). You must reduce the size of the prompt to continue.`);
-            }
-        
+
+            modelPromptMessagesML = truncatePromptIfNecessary(modelPromptMessagesML, tokenLength, this.getModelMaxTokenLength(), modelTargetTokenLength, pathwayResolver);
+
             const max_tokens = this.getModelMaxTokenLength() - tokenLength;
+            
+            if (max_tokens < 0) {
+                throw new Error(`Prompt is too long to successfully call the model at ${tokenLength} tokens.  The model will not be called.`);
+            }
         
             requestParameters = {
                 prompt: modelPromptMessagesML,
@@ -41,11 +55,14 @@ class OpenAICompletionPlugin extends ModelPlugin {
                 stream
             };
         } else {
-            if (tokenLength > modelTargetTokenLength) {
-                throw new Error(`Input is too long at ${tokenLength} tokens. The target token length for this pathway is ${modelTargetTokenLength} tokens because the response is expected to take up the rest of the ${this.getModelMaxTokenLength()} tokens that the model can handle. You must reduce the size of the prompt to continue.`);
-            }
-        
+
+            modelPromptText = truncatePromptIfNecessary(modelPromptText, tokenLength, this.getModelMaxTokenLength(), modelTargetTokenLength, pathwayResolver);
+
             const max_tokens = this.getModelMaxTokenLength() - tokenLength;
+            
+            if (max_tokens < 0) {
+                throw new Error(`Prompt is too long to successfully call the model at ${tokenLength} tokens.  The model will not be called.`);
+            }
         
             requestParameters = {
                 prompt: modelPromptText,
@@ -59,9 +76,9 @@ class OpenAICompletionPlugin extends ModelPlugin {
     }
 
     // Execute the request to the OpenAI Completion API
-    async execute(text, parameters, prompt) {
+    async execute(text, parameters, prompt, pathwayResolver) {
         const url = this.requestUrl(text);
-        const requestParameters = this.getRequestParameters(text, parameters, prompt);
+        const requestParameters = this.getRequestParameters(text, parameters, prompt, pathwayResolver);
     
         const data = { ...(this.model.params || {}), ...requestParameters };
         const params = {};