@aj-archipelago/cortex 1.1.33 → 1.1.35

package/README.md

@@ -432,7 +432,7 @@ Configuration of Cortex is done via a [convict](https://github.com/mozilla/node-
 - `PORT`: The port number for the Cortex server. Default is 4000. The value can be set using the `CORTEX_PORT` environment variable.
 - `storageConnectionString`: The connection string used for accessing storage. This is sensitive information and has no default value. The value can be set using the `STORAGE_CONNECTION_STRING` environment variable.
 
-The `buildPathways` function takes the config object and builds the `pathways` object by loading the core pathways and any custom pathways specified in the `pathwaysPath` property of the config object. The function returns the `pathways` object.
+The `buildPathways` function takes the config object and builds the `pathways` and `pathwayManager` objects by loading the core pathways and any custom pathways specified in the `pathwaysPath` property of the config object. The function returns the `pathways` and `pathwayManager` objects.
 
 The `buildModels` function takes the `config` object and builds the `models` object by compiling handlebars templates for each model specified in the `models` property of the config object. The function returns the `models` object.
 
@@ -471,3 +471,100 @@ Cortex is a constantly evolving project, and the following features are coming s
 * Prompt execution context preservation between calls (to enable interactive, multi-call integrations with LangChain and other technologies)
 * Model-specific cache key optimizations to increase hit rate and reduce cache size
 * Structured analytics and reporting on AI API call frequency, cost, cache hit rate, etc.
+
+## Dynamic Pathways
+
+Cortex supports dynamic pathways, which allow for the creation and management of pathways at runtime. This feature enables users to define custom pathways without modifying the core Cortex codebase.
+
+### How It Works
+
+1. Dynamic pathways are stored either locally or in cloud storage (Azure Blob Storage or AWS S3).
+2. The `PathwayManager` class handles loading, saving, and managing these dynamic pathways.
+3. Dynamic pathways can be added, updated, or removed via GraphQL mutations.
+
+### Configuration
+
+To use dynamic pathways, you need to provide a JSON configuration file or a JSON string. There are two ways to specify this configuration:
+
+1. Using a configuration file:
+   Set the `DYNAMIC_PATHWAYS_CONFIG_FILE` environment variable to the path of your JSON configuration file.
+
+2. Using a JSON string:
+   Set the `DYNAMIC_PATHWAYS_CONFIG_JSON` environment variable with the JSON configuration as a string.
+
+The configuration should include the following properties:
+
+```json
+{
+  "storageType": "local" | "azure" | "s3",
+  "filePath": "./dynamic/pathways.json",  // Only for local storage
+  "azureStorageConnectionString": "your_connection_string",  // Only for Azure
+  "azureContainerName": "cortexdynamicpathways",  // Optional, default is "cortexdynamicpathways"
+  "awsAccessKeyId": "your_access_key_id",  // Only for AWS S3
+  "awsSecretAccessKey": "your_secret_access_key",  // Only for AWS S3
+  "awsRegion": "your_aws_region",  // Only for AWS S3
+  "awsBucketName": "cortexdynamicpathways"  // Optional, default is "cortexdynamicpathways"
+}
+```
+
+### Storage Options
+
+1. Local Storage (default):
+   - Set `storageType` to `"local"`
+   - Specify `filePath` for the local JSON file (default: "./dynamic/pathways.json")
+
+2. Azure Blob Storage:
+   - Set `storageType` to `"azure"`
+   - Provide `azureStorageConnectionString`
+   - Optionally set `azureContainerName` (default: "cortexdynamicpathways")
+
+3. AWS S3:
+   - Set `storageType` to `"s3"`
+   - Provide `awsAccessKeyId`, `awsSecretAccessKey`, and `awsRegion`
+   - Optionally set `awsBucketName` (default: "cortexdynamicpathways")
+
+### Usage
+
+Dynamic pathways can be managed through GraphQL mutations. Here are the available operations:
+
+1. Adding or updating a pathway:
+
+```graphql
+mutation PutPathway($name: String!, $pathway: PathwayInput!, $userId: String!, $secret: String!, $displayName: String, $key: String!) {
+  putPathway(name: $name, pathway: $pathway, userId: $userId, secret: $secret, displayName: $displayName, key: $key) {
+    name
+  }
+}
+```
+
+2. Deleting a pathway:
+
+```graphql
+mutation DeletePathway($name: String!, $userId: String!, $secret: String!, $key: String!) {
+  deletePathway(name: $name, userId: $userId, secret: $secret, key: $key)
+}
+```
+
+3. Executing a dynamic pathway:
+
+```graphql
+query ExecuteWorkspace($userId: String!, $pathwayName: String!, $text: String!) {
+  executeWorkspace(userId: $userId, pathwayName: $pathwayName, text: $text) {
+    result
+  }
+}
+```
+
+### Security
+
+To ensure the security of dynamic pathways:
+
+1. A `PATHWAY_PUBLISH_KEY` environment variable must be set to enable pathway publishing.
+2. This key must be provided in the `key` parameter when adding, updating, or deleting pathways.
+3. Each pathway is associated with a `userId` and `secret`. The secret must be provided to modify or delete an existing pathway.
+
+### Synchronization across multiple instances
+
+Each instance of Cortex maintains its own local cache of pathways. On every dynamic pathway request, it checks if the local cache is up to date by comparing the last modified timestamp of the storage with the last update time of the local cache. If the local cache is out of date, it reloads the pathways from storage.
+
+This approach ensures that all instances of Cortex will eventually have access to the most up-to-date dynamic pathways without requiring immediate synchronization.

package/config/dynamicPathwaysConfig.example.json

@@ -0,0 +1,4 @@
+{
+    "storageType": "azure",
+    "filePath": "./pathways/dynamic/pathways.json",
+}

package/config.js

@@ -5,18 +5,19 @@ import fs from 'fs';
 import { fileURLToPath, pathToFileURL } from 'url';
 import GcpAuthTokenHelper from './lib/gcpAuthTokenHelper.js';
 import logger from './lib/logger.js';
+import PathwayManager from './lib/pathwayManager.js';
 
 const __dirname = path.dirname(fileURLToPath(import.meta.url));
 
 convict.addFormat({
     name: 'string-array',
-    validate: function(val) {
-      if (!Array.isArray(val)) {
-        throw new Error('must be of type Array');
-      }
+    validate: function (val) {
+        if (!Array.isArray(val)) {
+            throw new Error('must be of type Array');
+        }
     },
-    coerce: function(val) {
-      return val.split(',');
+    coerce: function (val) {
+        return val.split(',');
     },
 });
 
@@ -194,6 +195,13 @@ var config = convict({
                 "requestsPerSecond": 10,
                 "maxTokenLength": 200000
             },
+            "runware-flux-schnell": {
+                "type": "RUNWARE-AI",
+                "url": "https://api.runware.ai/v1",
+                "headers": {
+                    "Content-Type": "application/json"
+                },
+            },
         },
         env: 'CORTEX_MODELS'
     },
@@ -240,6 +248,12 @@ var config = convict({
         env: 'REDIS_ENCRYPTION_KEY',
         sensitive: true
     },
+    runwareAiApiKey: {
+        format: String,
+        default: null,
+        env: 'RUNWARE_API_KEY',
+        sensitive: true
+    },
     dalleImageApiUrl: {
         format: String,
         default: 'null',
@@ -290,6 +304,39 @@ if (config.get('gcpServiceAccountKey')) {
     config.set('gcpAuthTokenHelper', gcpAuthTokenHelper);
 }
 
+// Load dynamic pathways from JSON file or cloud storage
+const createDynamicPathwayManager = async (config, basePathway) => {
+    const { dynamicPathwayConfig } = config.getProperties();
+
+    if (!dynamicPathwayConfig) {
+        return null;
+    }
+
+    const storageConfig = {
+        storageType: dynamicPathwayConfig.storageType || 'local',
+        filePath: dynamicPathwayConfig.filePath || "./dynamic/pathways.json",
+        azureStorageConnectionString: dynamicPathwayConfig.azureStorageConnectionString,
+        azureContainerName: dynamicPathwayConfig.azureContainerName || 'cortexdynamicpathways',
+        awsAccessKeyId: dynamicPathwayConfig.awsAccessKeyId,
+        awsSecretAccessKey: dynamicPathwayConfig.awsSecretAccessKey,
+        awsRegion: dynamicPathwayConfig.awsRegion,
+        awsBucketName: dynamicPathwayConfig.awsBucketName || 'cortexdynamicpathways',
+    };
+
+    const pathwayManager = new PathwayManager(storageConfig, basePathway);
+
+    try {
+        const dynamicPathways = await pathwayManager.initialize();
+        logger.info(`Dynamic pathways loaded successfully`);
+        logger.info(`Loaded dynamic pathways for users: [${Object.keys(dynamicPathways).join(", ")}]`);
+
+        return pathwayManager;
+    } catch (error) {
+        logger.error(`Error loading dynamic pathways: ${error.message}`);
+        return pathwayManager;
+    }
+};
+
 // Build and load pathways to config
 const buildPathways = async (config) => {
     const { pathwaysPath, corePathwaysPath, basePathwayPath } = config.getProperties();
@@ -312,6 +359,32 @@ const buildPathways = async (config) => {
         loadedPathways = { ...loadedPathways, ...customPathways };
     }
 
+
+    const { DYNAMIC_PATHWAYS_CONFIG_FILE, DYNAMIC_PATHWAYS_CONFIG_JSON } = process.env;
+
+    let dynamicPathwayConfig;
+
+    // Load dynamic pathways
+    let pathwayManager;
+    try {
+        if (DYNAMIC_PATHWAYS_CONFIG_FILE) {
+            logger.info(`Reading dynamic pathway config from ${DYNAMIC_PATHWAYS_CONFIG_FILE}`);
+            dynamicPathwayConfig = JSON.parse(fs.readFileSync(DYNAMIC_PATHWAYS_CONFIG_FILE, 'utf8'));
+        } else if (DYNAMIC_PATHWAYS_CONFIG_JSON) {
+            logger.info(`Reading dynamic pathway config from DYNAMIC_PATHWAYS_CONFIG_JSON variable`);
+            dynamicPathwayConfig = JSON.parse(DYNAMIC_PATHWAYS_CONFIG_JSON);
+        }
+        else {
+            logger.warn('Dynamic pathways are not enabled. Please set the DYNAMIC_PATHWAYS_CONFIG_FILE or DYNAMIC_PATHWAYS_CONFIG_JSON environment variable to enable dynamic pathways.');
+        }
+
+        config.load({ dynamicPathwayConfig });
+        pathwayManager = await createDynamicPathwayManager(config, basePathway);
+    } catch (error) {
+        logger.error(`Error loading dynamic pathways: ${error.message}`);
+        process.exit(1);
+    }
+
     // This is where we integrate pathway overrides from the config
     // file. This can run into a partial definition issue if the
     // config file contains pathways that no longer exist.
@@ -322,9 +395,9 @@ const buildPathways = async (config) => {
     }
 
     // Add pathways to config
-    config.load({ pathways })
+    config.load({ pathways });
 
-    return pathways;
+    return { pathwayManager, pathways };
 }
 
 // Build and load models to config
@@ -336,7 +409,7 @@ const buildModels = (config) => {
         if (!model.name) {
             model.name = key;
         }
-        
+
         // if model is in old format, convert it to new format
         if (!model.endpoints) {
             model = {
@@ -354,7 +427,7 @@ const buildModels = (config) => {
         }
 
         // compile handlebars templates for each endpoint
-        model.endpoints = model.endpoints.map(endpoint => 
+        model.endpoints = model.endpoints.map(endpoint =>
             JSON.parse(HandleBars.compile(JSON.stringify(endpoint))({ ...model, ...config.getEnv(), ...config.getProperties() }))
         );
 

package/helper-apps/cortex-autogen/function_app.py

@@ -5,6 +5,13 @@ from azure.storage.queue import QueueClient
 import os
 import redis
 from myautogen import process_message
+import subprocess
+import sys
+
+def install_packages():
+    subprocess.check_call([sys.executable, "-m", "pip", "install", "-r", "requirements.txt"])
+
+install_packages()
 
 app = func.FunctionApp()
 

package/helper-apps/cortex-autogen/myautogen.py

@@ -12,11 +12,28 @@ import requests
 import pathlib
 import pymongo
 import logging
-from datetime import datetime, timezone
-from tools.sasfileuploader import autogen_sas_uploader
+from datetime import datetime, timezone, timedelta
 import shutil
+import time
+import base64
+import zipfile
+from azure.storage.blob import BlobServiceClient, generate_blob_sas, BlobSasPermissions
+
 load_dotenv()
 
+connection_string = os.environ["AZURE_STORAGE_CONNECTION_STRING"]
+human_input_queue_name = os.environ.get("HUMAN_INPUT_QUEUE_NAME", "autogen-human-input-queue")
+human_input_queue_client = QueueClient.from_connection_string(connection_string, human_input_queue_name)
+
+def check_for_human_input(request_id):
+    messages = human_input_queue_client.receive_messages()
+    for message in messages:
+        content = json.loads(base64.b64decode(message.content).decode('utf-8'))
+        if content['codeRequestId'] == request_id:
+            human_input_queue_client.delete_message(message)
+            return content['text']
+    return None
+
 DEFAULT_SUMMARY_PROMPT = "Summarize the takeaway from the conversation. Do not add any introductory phrases."
 try:
     with open("prompt_summary.txt", "r") as file:
@@ -36,12 +53,6 @@ def store_in_mongo(data):
     except Exception as e:
         logging.error(f"An error occurred while storing data in MongoDB: {str(e)}")
 
-app = func.FunctionApp()
-
-connection_string = os.environ["AZURE_STORAGE_CONNECTION_STRING"]
-queue_name = os.environ.get("QUEUE_NAME", "autogen-message-queue")
-queue_client = QueueClient.from_connection_string(connection_string, queue_name)
-
 redis_client = redis.from_url(os.environ['REDIS_CONNECTION_STRING'])
 channel = 'requestProgress'
 
@@ -58,7 +69,7 @@ def publish_request_progress(data):
     if connect_redis():
         try:
             message = json.dumps(data)
-            logging.info(f"Publishing message {message} to channel {channel}")
+            #logging.info(f"Publishing message {message} to channel {channel}")
             redis_client.publish(channel, message)
         except Exception as e:
             logging.error(f"Error publishing message: {e}")
@@ -95,8 +106,41 @@ def fetch_from_url(url):
         logging.error(f"Error fetching from URL: {e}")
         return ""
 
+
+def zip_and_upload_tmp_folder(temp_dir):
+    zip_path = os.path.join(temp_dir, "tmp_contents.zip")
+    with zipfile.ZipFile(zip_path, 'w', zipfile.ZIP_DEFLATED) as zipf:
+        for root, _, files in os.walk(temp_dir):
+            for file in files:
+                file_path = os.path.join(root, file)
+                arcname = os.path.relpath(file_path, temp_dir)
+                zipf.write(file_path, arcname)
+
+    blob_service_client = BlobServiceClient.from_connection_string(os.environ["AZURE_STORAGE_CONNECTION_STRING"])
+    container_name = os.environ.get("AZURE_BLOB_CONTAINER", "autogen-uploads")
+    blob_name = f"tmp_contents_{datetime.now(timezone.utc).strftime('%Y%m%d%H%M%S')}.zip"
+    blob_client = blob_service_client.get_blob_client(container=container_name, blob=blob_name)
+
+    with open(zip_path, "rb") as data:
+        blob_client.upload_blob(data)
+
+    account_key = blob_service_client.credential.account_key
+    account_name = blob_service_client.account_name
+    expiry = datetime.now(timezone.utc) + timedelta(hours=1)
+
+    sas_token = generate_blob_sas(
+        account_name,
+        container_name,
+        blob_name,
+        account_key=account_key,
+        permission=BlobSasPermissions(read=True),
+        expiry=expiry
+    )
+
+    return f"{blob_client.url}?{sas_token}"
+
 def process_message(message_data, original_request_message):
-    logging.info(f"Processing Message: {message_data}")
+    # logging.info(f"Processing Message: {message_data}")
     try:
         started_at = datetime.now()
         message = message_data['message']
@@ -117,14 +161,15 @@ def process_message(message_data, original_request_message):
             total_messages = 20 * 2
             all_messages = []
 
+            terminate_count = 0
             def is_termination_msg(m):
-                content = m.get("content", "")
-                if message_count == 0:
+                nonlocal terminate_count
+                content = m.get("content", "").strip()
+                if not content:
                     return False
-                return (m.get("role") == "assistant" and not content.strip()) or \
-                    content.rstrip().endswith("TERMINATE") or \
-                    "first message must use the" in content.lower() or \
-                    len(content.strip()) == 0
+                if content.rstrip().endswith("TERMINATE"):
+                    terminate_count += 1
+                return terminate_count >= 3 or "first message must use the" in content.lower()
 
             system_message_given = get_given_system_message()
             system_message_assistant = AssistantAgent.DEFAULT_SYSTEM_MESSAGE 
@@ -137,17 +182,18 @@ def process_message(message_data, original_request_message):
             assistant = AssistantAgent("assistant", 
                 llm_config=llm_config, 
                 system_message=system_message_assistant,
-                code_execution_config={"executor": code_executor},
+                # code_execution_config={"executor": code_executor},
                 is_termination_msg=is_termination_msg,
             )
             
             user_proxy = UserProxyAgent(
                 "user_proxy",
-                llm_config=llm_config,
+                # llm_config=llm_config,
                 system_message=system_message_given,
                 code_execution_config={"executor": code_executor},
                 human_input_mode="NEVER",
                 max_consecutive_auto_reply=20,
+                is_termination_msg=is_termination_msg,
             )
 
             # description = "Upload a file to Azure Blob Storage and get URL back with a SAS token. Requires AZURE_STORAGE_CONNECTION_STRING and AZURE_BLOB_CONTAINER environment variables. Input: file_path (str). Output: SAS URL (str) or error message."
@@ -175,10 +221,35 @@ def process_message(message_data, original_request_message):
                 nonlocal message_count, all_messages
                 if not message:
                     return
+                
+                if True or sender.name == "user_proxy":
+                    human_input = check_for_human_input(request_id)
+                    if human_input:
+                        if human_input == "TERMINATE":
+                            logging.info("Terminating conversation")
+                            raise Exception("Conversation terminated by user")
+                        elif human_input == "PAUSE":
+                            logging.info("Pausing conversation")
+                            pause_start = time.time()
+                            while time.time() - pause_start < 60*15:  # 15 minutes pause timeout
+                                time.sleep(10)
+                                new_input = check_for_human_input(request_id)
+                                if new_input:
+                                    logging.info(f"Resuming conversation with human input: {new_input}")
+                                    return logged_send(sender, original_send, new_input, recipient, request_reply, silent)
+                            logging.info("Pause timeout, ending conversation")
+                            raise Exception("Conversation ended due to pause timeout")
+                        logging.info(f"Human input to {recipient.name}: {human_input}")
+                        return original_send(human_input, recipient, request_reply, silent)
+
+
                 logging.info(f"Message from {sender.name} to {recipient.name}: {message}")
+
                 message_count += 1
                 progress = min(message_count / total_messages, 1)
                 all_messages.append({"sender": sender.name, "message": message})
+
+                # if sender.name == "assistant":
                 publish_request_progress({
                     "requestId": request_id,
                     "progress": progress,
@@ -192,6 +263,9 @@ def process_message(message_data, original_request_message):
             #summary_method="reflection_with_llm", "last_msg"
             chat_result = user_proxy.initiate_chat(assistant, message=message, summary_method="reflection_with_llm", summary_args={"summary_role": "user", "summary_prompt": summary_prompt})
 
+
+            zip_url = zip_and_upload_tmp_folder(temp_dir)
+
             msg = ""
             try:
                 msg = all_messages[-1 if all_messages[-2]["message"] else -3]["message"] 
@@ -201,6 +275,7 @@ def process_message(message_data, original_request_message):
                 msg = f"Finished, with errors 🤖 ... {e}"
 
             msg = chat_result.summary if chat_result.summary else msg
+            msg += f"\n\n[Download all files of this task]({zip_url})"
 
             finalData = {
                 "requestId": request_id,
@@ -224,5 +299,19 @@ def process_message(message_data, original_request_message):
             publish_request_progress({
                 "requestId": request_id,
                 "progress": 1,
-                "error": str(e)
-            })
+                "error": str(e),
+                "data": str(e),
+            })
+            store_in_mongo({
+                "requestId": request_id,
+                "requestMessage": message_data.get("message"),
+                "progress": 1,
+                "error": str(e),
+                "data": str(e),
+                "contextId": message_data.get("contextId"),
+                "conversation": all_messages,
+                "createdAt": datetime.now(timezone.utc).isoformat(),
+                "insertionTime": original_request_message.insertion_time.astimezone(timezone.utc).isoformat() if original_request_message else None,
+                "startedAt": started_at.astimezone(timezone.utc).isoformat(),
+            })
+

package/helper-apps/cortex-autogen/prompt_summary.txt

@@ -4,10 +4,6 @@ Avoid expressing gratitude or using pleasantries.
 Maintain a professional and direct tone throughout responses.
 Include most recent meaningful messages from the conversation in the summary.
 You must include all your uploaded URLs, and url of your uploaded final code URL.
-Reply must be in markdown format, including images and videos as UI can show markdown directly to user in a nice way, so make sure to include all visuals, you may do as follows:
-    For images: ![Alt Text](IMAGE_URL)
-    For videos: <video src="VIDEO_URL" controls></video>
-    For urls: [Link Text](URL)
 Your reply will be only thing that finally gets to surface so make sure it is complete.
 Do not mention words like "Summary of the conversation", "Response", "Task", "The conversation" or so as it doesn't makes sense. 
 Also no need for "Request", user already know its request and task.
@@ -24,5 +20,18 @@ No need to say none of this as user already 'll be aware as has got the result:
     - Script executed twice due to debugging environment ...
     - Verification code ...
     - Issues encountered and resolved: ...
+    - The original plan ...
+    - Performed at ...
 
+No need to mention about code files uploaded to Azure Blob or point URLs as SAS-URLS as its a task that you already do and is known.
+No need to mention SAS URL, just give the url itself.
+Never include TERMINATE in your response.
 
+When formulating your responses, it's crucial to leverage the full capabilities of markdown formatting to create rich, visually appealing content. This approach not only enhances the user experience but also allows for more effective communication of complex ideas. Follow these guidelines to ensure your responses are both informative and visually engaging, create responses that are not only informative but also visually appealing and easy to comprehend:
+    - For images: ![Alt Text](IMAGE_URL) and <img src="IMAGE_URL" alt="Alt Text">
+    - For videos: <video src="VIDEO_URL" controls></video>
+    - For urls: [Link Text](URL)
+If there an image url, you must always include it as url and markdown e.g.: ![Alt Text](IMAGE_URL) and [Alt Text](IMAGE_URL)
+
+
+Make sure to present it nicely so human finds it appealing.

package/helper-apps/cortex-autogen/requirements.txt

@@ -1,6 +1,6 @@
 azure-storage-queue
 azure-functions
-pyautogen
+pyautogen==0.3.0
 redis
 pymongo
 requests