@aj-archipelago/cortex 1.1.32 → 1.1.34

package/README.md

@@ -432,7 +432,7 @@ Configuration of Cortex is done via a [convict](https://github.com/mozilla/node-
 - `PORT`: The port number for the Cortex server. Default is 4000. The value can be set using the `CORTEX_PORT` environment variable.
 - `storageConnectionString`: The connection string used for accessing storage. This is sensitive information and has no default value. The value can be set using the `STORAGE_CONNECTION_STRING` environment variable.
 
-The `buildPathways` function takes the config object and builds the `pathways` object by loading the core pathways and any custom pathways specified in the `pathwaysPath` property of the config object. The function returns the `pathways` object.
+The `buildPathways` function takes the config object and builds the `pathways` and `pathwayManager` objects by loading the core pathways and any custom pathways specified in the `pathwaysPath` property of the config object. The function returns the `pathways` and `pathwayManager` objects.
 
 The `buildModels` function takes the `config` object and builds the `models` object by compiling handlebars templates for each model specified in the `models` property of the config object. The function returns the `models` object.
 
@@ -471,3 +471,100 @@ Cortex is a constantly evolving project, and the following features are coming s
 * Prompt execution context preservation between calls (to enable interactive, multi-call integrations with LangChain and other technologies)
 * Model-specific cache key optimizations to increase hit rate and reduce cache size
 * Structured analytics and reporting on AI API call frequency, cost, cache hit rate, etc.
+
+## Dynamic Pathways
+
+Cortex supports dynamic pathways, which allow for the creation and management of pathways at runtime. This feature enables users to define custom pathways without modifying the core Cortex codebase.
+
+### How It Works
+
+1. Dynamic pathways are stored either locally or in cloud storage (Azure Blob Storage or AWS S3).
+2. The `PathwayManager` class handles loading, saving, and managing these dynamic pathways.
+3. Dynamic pathways can be added, updated, or removed via GraphQL mutations.
+
+### Configuration
+
+To use dynamic pathways, you need to provide a JSON configuration file or a JSON string. There are two ways to specify this configuration:
+
+1. Using a configuration file:
+   Set the `DYNAMIC_PATHWAYS_CONFIG_FILE` environment variable to the path of your JSON configuration file.
+
+2. Using a JSON string:
+   Set the `DYNAMIC_PATHWAYS_CONFIG_JSON` environment variable with the JSON configuration as a string.
+
+The configuration should include the following properties:
+
+```json
+{
+  "storageType": "local" | "azure" | "s3",
+  "filePath": "./dynamic/pathways.json",  // Only for local storage
+  "azureStorageConnectionString": "your_connection_string",  // Only for Azure
+  "azureContainerName": "cortexdynamicpathways",  // Optional, default is "cortexdynamicpathways"
+  "awsAccessKeyId": "your_access_key_id",  // Only for AWS S3
+  "awsSecretAccessKey": "your_secret_access_key",  // Only for AWS S3
+  "awsRegion": "your_aws_region",  // Only for AWS S3
+  "awsBucketName": "cortexdynamicpathways"  // Optional, default is "cortexdynamicpathways"
+}
+```
+
+### Storage Options
+
+1. Local Storage (default):
+   - Set `storageType` to `"local"`
+   - Specify `filePath` for the local JSON file (default: "./dynamic/pathways.json")
+
+2. Azure Blob Storage:
+   - Set `storageType` to `"azure"`
+   - Provide `azureStorageConnectionString`
+   - Optionally set `azureContainerName` (default: "cortexdynamicpathways")
+
+3. AWS S3:
+   - Set `storageType` to `"s3"`
+   - Provide `awsAccessKeyId`, `awsSecretAccessKey`, and `awsRegion`
+   - Optionally set `awsBucketName` (default: "cortexdynamicpathways")
+
+### Usage
+
+Dynamic pathways can be managed through GraphQL mutations. Here are the available operations:
+
+1. Adding or updating a pathway:
+
+```graphql
+mutation PutPathway($name: String!, $pathway: PathwayInput!, $userId: String!, $secret: String!, $displayName: String, $key: String!) {
+  putPathway(name: $name, pathway: $pathway, userId: $userId, secret: $secret, displayName: $displayName, key: $key) {
+    name
+  }
+}
+```
+
+2. Deleting a pathway:
+
+```graphql
+mutation DeletePathway($name: String!, $userId: String!, $secret: String!, $key: String!) {
+  deletePathway(name: $name, userId: $userId, secret: $secret, key: $key)
+}
+```
+
+3. Executing a dynamic pathway:
+
+```graphql
+query ExecuteWorkspace($userId: String!, $pathwayName: String!, $text: String!) {
+  executeWorkspace(userId: $userId, pathwayName: $pathwayName, text: $text) {
+    result
+  }
+}
+```
+
+### Security
+
+To ensure the security of dynamic pathways:
+
+1. A `PATHWAY_PUBLISH_KEY` environment variable must be set to enable pathway publishing.
+2. This key must be provided in the `key` parameter when adding, updating, or deleting pathways.
+3. Each pathway is associated with a `userId` and `secret`. The secret must be provided to modify or delete an existing pathway.
+
+### Synchronization across multiple instances
+
+Each instance of Cortex maintains its own local cache of pathways. On every dynamic pathway request, it checks if the local cache is up to date by comparing the last modified timestamp of the storage with the last update time of the local cache. If the local cache is out of date, it reloads the pathways from storage.
+
+This approach ensures that all instances of Cortex will eventually have access to the most up-to-date dynamic pathways without requiring immediate synchronization.

package/config/dynamicPathwaysConfig.example.json

@@ -0,0 +1,4 @@
+{
+    "storageType": "azure",
+    "filePath": "./pathways/dynamic/pathways.json",
+}

package/config.js

@@ -5,18 +5,19 @@ import fs from 'fs';
 import { fileURLToPath, pathToFileURL } from 'url';
 import GcpAuthTokenHelper from './lib/gcpAuthTokenHelper.js';
 import logger from './lib/logger.js';
+import PathwayManager from './lib/pathwayManager.js';
 
 const __dirname = path.dirname(fileURLToPath(import.meta.url));
 
 convict.addFormat({
     name: 'string-array',
-    validate: function(val) {
-      if (!Array.isArray(val)) {
-        throw new Error('must be of type Array');
-      }
+    validate: function (val) {
+        if (!Array.isArray(val)) {
+            throw new Error('must be of type Array');
+        }
     },
-    coerce: function(val) {
-      return val.split(',');
+    coerce: function (val) {
+        return val.split(',');
     },
 });
 
@@ -194,6 +195,13 @@ var config = convict({
                 "requestsPerSecond": 10,
                 "maxTokenLength": 200000
             },
+            "runware-flux-schnell": {
+                "type": "RUNWARE-AI",
+                "url": "https://api.runware.ai/v1",
+                "headers": {
+                    "Content-Type": "application/json"
+                },
+            },
         },
         env: 'CORTEX_MODELS'
     },
@@ -240,6 +248,12 @@ var config = convict({
         env: 'REDIS_ENCRYPTION_KEY',
         sensitive: true
     },
+    runwareAiApiKey: {
+        format: String,
+        default: null,
+        env: 'RUNWARE_API_KEY',
+        sensitive: true
+    },
     dalleImageApiUrl: {
         format: String,
         default: 'null',
@@ -290,6 +304,39 @@ if (config.get('gcpServiceAccountKey')) {
     config.set('gcpAuthTokenHelper', gcpAuthTokenHelper);
 }
 
+// Load dynamic pathways from JSON file or cloud storage
+const createDynamicPathwayManager = async (config, basePathway) => {
+    const { dynamicPathwayConfig } = config.getProperties();
+
+    if (!dynamicPathwayConfig) {
+        return null;
+    }
+
+    const storageConfig = {
+        storageType: dynamicPathwayConfig.storageType || 'local',
+        filePath: dynamicPathwayConfig.filePath || "./dynamic/pathways.json",
+        azureStorageConnectionString: dynamicPathwayConfig.azureStorageConnectionString,
+        azureContainerName: dynamicPathwayConfig.azureContainerName || 'cortexdynamicpathways',
+        awsAccessKeyId: dynamicPathwayConfig.awsAccessKeyId,
+        awsSecretAccessKey: dynamicPathwayConfig.awsSecretAccessKey,
+        awsRegion: dynamicPathwayConfig.awsRegion,
+        awsBucketName: dynamicPathwayConfig.awsBucketName || 'cortexdynamicpathways',
+    };
+
+    const pathwayManager = new PathwayManager(storageConfig, basePathway);
+
+    try {
+        const dynamicPathways = await pathwayManager.initialize();
+        logger.info(`Dynamic pathways loaded successfully`);
+        logger.info(`Loaded dynamic pathways for users: [${Object.keys(dynamicPathways).join(", ")}]`);
+
+        return pathwayManager;
+    } catch (error) {
+        logger.error(`Error loading dynamic pathways: ${error.message}`);
+        return pathwayManager;
+    }
+};
+
 // Build and load pathways to config
 const buildPathways = async (config) => {
     const { pathwaysPath, corePathwaysPath, basePathwayPath } = config.getProperties();
@@ -312,6 +359,32 @@ const buildPathways = async (config) => {
         loadedPathways = { ...loadedPathways, ...customPathways };
     }
 
+
+    const { DYNAMIC_PATHWAYS_CONFIG_FILE, DYNAMIC_PATHWAYS_CONFIG_JSON } = process.env;
+
+    let dynamicPathwayConfig;
+
+    // Load dynamic pathways
+    let pathwayManager;
+    try {
+        if (DYNAMIC_PATHWAYS_CONFIG_FILE) {
+            logger.info(`Reading dynamic pathway config from ${DYNAMIC_PATHWAYS_CONFIG_FILE}`);
+            dynamicPathwayConfig = JSON.parse(fs.readFileSync(DYNAMIC_PATHWAYS_CONFIG_FILE, 'utf8'));
+        } else if (DYNAMIC_PATHWAYS_CONFIG_JSON) {
+            logger.info(`Reading dynamic pathway config from DYNAMIC_PATHWAYS_CONFIG_JSON variable`);
+            dynamicPathwayConfig = JSON.parse(DYNAMIC_PATHWAYS_CONFIG_JSON);
+        }
+        else {
+            logger.warn('Dynamic pathways are not enabled. Please set the DYNAMIC_PATHWAYS_CONFIG_FILE or DYNAMIC_PATHWAYS_CONFIG_JSON environment variable to enable dynamic pathways.');
+        }
+
+        config.load({ dynamicPathwayConfig });
+        pathwayManager = await createDynamicPathwayManager(config, basePathway);
+    } catch (error) {
+        logger.error(`Error loading dynamic pathways: ${error.message}`);
+        process.exit(1);
+    }
+
     // This is where we integrate pathway overrides from the config
     // file. This can run into a partial definition issue if the
     // config file contains pathways that no longer exist.
@@ -322,9 +395,9 @@ const buildPathways = async (config) => {
     }
 
     // Add pathways to config
-    config.load({ pathways })
+    config.load({ pathways });
 
-    return pathways;
+    return { pathwayManager, pathways };
 }
 
 // Build and load models to config
@@ -336,7 +409,7 @@ const buildModels = (config) => {
         if (!model.name) {
             model.name = key;
         }
-        
+
         // if model is in old format, convert it to new format
         if (!model.endpoints) {
             model = {
@@ -354,7 +427,7 @@ const buildModels = (config) => {
         }
 
         // compile handlebars templates for each endpoint
-        model.endpoints = model.endpoints.map(endpoint => 
+        model.endpoints = model.endpoints.map(endpoint =>
             JSON.parse(HandleBars.compile(JSON.stringify(endpoint))({ ...model, ...config.getEnv(), ...config.getProperties() }))
         );
 

package/helper-apps/cortex-autogen/function_app.py

@@ -1,13 +1,17 @@
 import azure.functions as func
 import logging
 import json
-import autogen
-from autogen import AssistantAgent, UserProxyAgent, config_list_from_json
 from azure.storage.queue import QueueClient
 import os
-import tempfile
 import redis
 from myautogen import process_message
+import subprocess
+import sys
+
+def install_packages():
+    subprocess.check_call([sys.executable, "-m", "pip", "install", "-r", "requirements.txt"])
+
+install_packages()
 
 app = func.FunctionApp()
 
@@ -26,7 +30,7 @@ def queue_trigger(msg: func.QueueMessage):
         message_data = json.loads(msg.get_body().decode('utf-8'))
         if "requestId" not in message_data:
             message_data['requestId'] = msg.id
-        process_message(message_data)
+        process_message(message_data, msg)
 
     except Exception as e:
         logging.error(f"Error processing message: {str(e)}")

package/helper-apps/cortex-autogen/main.py

@@ -25,7 +25,7 @@ def main():
                 message_data = json.loads(decoded_content)
                 if "requestId" not in message_data:
                     message_data['requestId'] = message.id
-                process_message(message_data)
+                process_message(message_data, message)
                 queue_client.delete_message(message)
             attempts = 0  # Reset attempts if a message was processed
         else:

package/helper-apps/cortex-autogen/myautogen.py

@@ -2,7 +2,7 @@ import azure.functions as func
 import logging
 import json
 import autogen
-from autogen import AssistantAgent, UserProxyAgent, config_list_from_json
+from autogen import AssistantAgent, UserProxyAgent, config_list_from_json, register_function
 from azure.storage.queue import QueueClient
 import os
 import tempfile
@@ -10,14 +10,48 @@ import redis
 from dotenv import load_dotenv
 import requests
 import pathlib
+import pymongo
+import logging
+from datetime import datetime, timezone, timedelta
+import shutil
+import time
+import base64
+import zipfile
+from azure.storage.blob import BlobServiceClient, generate_blob_sas, BlobSasPermissions
 
 load_dotenv()
 
-app = func.FunctionApp()
-
 connection_string = os.environ["AZURE_STORAGE_CONNECTION_STRING"]
-queue_name = os.environ.get("QUEUE_NAME", "autogen-message-queue")
-queue_client = QueueClient.from_connection_string(connection_string, queue_name)
+human_input_queue_name = os.environ.get("HUMAN_INPUT_QUEUE_NAME", "autogen-human-input-queue")
+human_input_queue_client = QueueClient.from_connection_string(connection_string, human_input_queue_name)
+
+def check_for_human_input(request_id):
+    messages = human_input_queue_client.receive_messages()
+    for message in messages:
+        content = json.loads(base64.b64decode(message.content).decode('utf-8'))
+        if content['codeRequestId'] == request_id:
+            human_input_queue_client.delete_message(message)
+            return content['text']
+    return None
+
+DEFAULT_SUMMARY_PROMPT = "Summarize the takeaway from the conversation. Do not add any introductory phrases."
+try:
+    with open("prompt_summary.txt", "r") as file:
+        summary_prompt = file.read() or DEFAULT_SUMMARY_PROMPT
+except FileNotFoundError:
+    summary_prompt = DEFAULT_SUMMARY_PROMPT 
+
+
+def store_in_mongo(data):
+    try:
+        if 'MONGO_URI' in os.environ:
+            client = pymongo.MongoClient(os.environ['MONGO_URI'])
+            collection = client.get_default_database()[os.environ.get('MONGO_COLLECTION_NAME', 'autogenruns')]
+            collection.insert_one(data)
+        else:
+            logging.warning("MONGO_URI not found in environment variables")
+    except Exception as e:
+        logging.error(f"An error occurred while storing data in MongoDB: {str(e)}")
 
 redis_client = redis.from_url(os.environ['REDIS_CONNECTION_STRING'])
 channel = 'requestProgress'
@@ -35,7 +69,7 @@ def publish_request_progress(data):
     if connect_redis():
         try:
             message = json.dumps(data)
-            logging.info(f"Publishing message {message} to channel {channel}")
+            #logging.info(f"Publishing message {message} to channel {channel}")
             redis_client.publish(channel, message)
         except Exception as e:
             logging.error(f"Error publishing message: {e}")
@@ -72,45 +106,89 @@ def fetch_from_url(url):
         logging.error(f"Error fetching from URL: {e}")
         return ""
 
-def process_message(message_data):
-    logging.info(f"Processing Message: {message_data}")
+
+def zip_and_upload_tmp_folder(temp_dir):
+    zip_path = os.path.join(temp_dir, "tmp_contents.zip")
+    with zipfile.ZipFile(zip_path, 'w', zipfile.ZIP_DEFLATED) as zipf:
+        for root, _, files in os.walk(temp_dir):
+            for file in files:
+                file_path = os.path.join(root, file)
+                arcname = os.path.relpath(file_path, temp_dir)
+                zipf.write(file_path, arcname)
+
+    blob_service_client = BlobServiceClient.from_connection_string(os.environ["AZURE_STORAGE_CONNECTION_STRING"])
+    container_name = os.environ.get("AZURE_BLOB_CONTAINER", "autogen-uploads")
+    blob_name = f"tmp_contents_{datetime.now(timezone.utc).strftime('%Y%m%d%H%M%S')}.zip"
+    blob_client = blob_service_client.get_blob_client(container=container_name, blob=blob_name)
+
+    with open(zip_path, "rb") as data:
+        blob_client.upload_blob(data)
+
+    account_key = blob_service_client.credential.account_key
+    account_name = blob_service_client.account_name
+    expiry = datetime.now(timezone.utc) + timedelta(hours=1)
+
+    sas_token = generate_blob_sas(
+        account_name,
+        container_name,
+        blob_name,
+        account_key=account_key,
+        permission=BlobSasPermissions(read=True),
+        expiry=expiry
+    )
+
+    return f"{blob_client.url}?{sas_token}"
+
+def process_message(message_data, original_request_message):
+    # logging.info(f"Processing Message: {message_data}")
     try:
+        started_at = datetime.now()
         message = message_data['message']
         request_id = message_data.get('requestId') or msg.id
 
         config_list = config_list_from_json(env_or_file="OAI_CONFIG_LIST")
         base_url = os.environ.get("CORTEX_API_BASE_URL")
         api_key = os.environ.get("CORTEX_API_KEY")
-        llm_config = {"config_list": config_list, "base_url": base_url, "api_key": api_key, "cache_seed": None}
+        llm_config = {"config_list": config_list, "base_url": base_url, "api_key": api_key, "cache_seed": None, "timeout": 600}
 
         with tempfile.TemporaryDirectory() as temp_dir:
+            #copy /tools directory to temp_dir
+            shutil.copytree(os.path.join(os.getcwd(), "tools"), temp_dir, dirs_exist_ok=True)
+            
             code_executor = autogen.coding.LocalCommandLineCodeExecutor(work_dir=temp_dir)
 
             message_count = 0
             total_messages = 20 * 2
             all_messages = []
 
+            terminate_count = 0
             def is_termination_msg(m):
-                content = m.get("content", "")
-                if message_count == 0:
+                nonlocal terminate_count
+                content = m.get("content", "").strip()
+                if not content:
                     return False
-                return (m.get("role") == "assistant" and not content.strip()) or \
-                    content.rstrip().endswith("TERMINATE") or \
-                    "first message must use the" in content.lower() or \
-                    len(content.strip()) == 0
+                if content.rstrip().endswith("TERMINATE"):
+                    terminate_count += 1
+                return terminate_count >= 3 or "first message must use the" in content.lower()
 
             system_message_given = get_given_system_message()
             system_message_assistant = AssistantAgent.DEFAULT_SYSTEM_MESSAGE 
 
             if system_message_given:
-                system_message_assistant = f"{system_message_assistant}\n\n{system_message_given}"
+                system_message_assistant = system_message_given
             else:
                 print("No extra system message given for assistant")
 
-            assistant = AssistantAgent("assistant", llm_config=llm_config, system_message=system_message_assistant)
-
+            assistant = AssistantAgent("assistant", 
+                llm_config=llm_config, 
+                system_message=system_message_assistant,
+                # code_execution_config={"executor": code_executor},
+                is_termination_msg=is_termination_msg,
+            )
+            
             user_proxy = UserProxyAgent(
                 "user_proxy",
+                # llm_config=llm_config,
                 system_message=system_message_given,
                 code_execution_config={"executor": code_executor},
                 human_input_mode="NEVER",
@@ -118,15 +196,60 @@ def process_message(message_data):
                 is_termination_msg=is_termination_msg,
             )
 
+            # description = "Upload a file to Azure Blob Storage and get URL back with a SAS token. Requires AZURE_STORAGE_CONNECTION_STRING and AZURE_BLOB_CONTAINER environment variables. Input: file_path (str). Output: SAS URL (str) or error message."
+
+            # register_function(
+            #     autogen_sas_uploader,
+            #     caller=assistant,
+            #     executor=user_proxy,
+            #     name="autogen_sas_uploader",
+            #     description=description,    
+            # )
+
+            # register_function(
+            #     autogen_sas_uploader,
+            #     caller=user_proxy,
+            #     executor=assistant,
+            #     name="autogen_sas_uploader",
+            #     description=description,    
+            # )
+
             original_assistant_send = assistant.send
             original_user_proxy_send = user_proxy.send
 
             def logged_send(sender, original_send, message, recipient, request_reply=None, silent=True):
                 nonlocal message_count, all_messages
+                if not message:
+                    return
+                
+                if True or sender.name == "user_proxy":
+                    human_input = check_for_human_input(request_id)
+                    if human_input:
+                        if human_input == "TERMINATE":
+                            logging.info("Terminating conversation")
+                            raise Exception("Conversation terminated by user")
+                        elif human_input == "PAUSE":
+                            logging.info("Pausing conversation")
+                            pause_start = time.time()
+                            while time.time() - pause_start < 60*15:  # 15 minutes pause timeout
+                                time.sleep(10)
+                                new_input = check_for_human_input(request_id)
+                                if new_input:
+                                    logging.info(f"Resuming conversation with human input: {new_input}")
+                                    return logged_send(sender, original_send, new_input, recipient, request_reply, silent)
+                            logging.info("Pause timeout, ending conversation")
+                            raise Exception("Conversation ended due to pause timeout")
+                        logging.info(f"Human input to {recipient.name}: {human_input}")
+                        return original_send(human_input, recipient, request_reply, silent)
+
+
                 logging.info(f"Message from {sender.name} to {recipient.name}: {message}")
+
                 message_count += 1
                 progress = min(message_count / total_messages, 1)
                 all_messages.append({"sender": sender.name, "message": message})
+
+                # if sender.name == "assistant":
                 publish_request_progress({
                     "requestId": request_id,
                     "progress": progress,
@@ -134,19 +257,41 @@ def process_message(message_data):
                 })
                 return original_send(message, recipient, request_reply, silent)
 
-            assistant.send = lambda message, recipient, request_reply=None, silent=True: logged_send(assistant, original_assistant_send, message, recipient, request_reply, silent)
-            user_proxy.send = lambda message, recipient, request_reply=None, silent=True: logged_send(user_proxy, original_user_proxy_send, message, recipient, request_reply, silent)
+            assistant.send = lambda message, recipient, request_reply=None, silent=False: logged_send(assistant, original_assistant_send, message, recipient, request_reply, silent)
+            user_proxy.send = lambda message, recipient, request_reply=None, silent=False: logged_send(user_proxy, original_user_proxy_send, message, recipient, request_reply, silent)
 
-            chat_result = user_proxy.initiate_chat(assistant, message=message)
+            #summary_method="reflection_with_llm", "last_msg"
+            chat_result = user_proxy.initiate_chat(assistant, message=message, summary_method="reflection_with_llm", summary_args={"summary_role": "user", "summary_prompt": summary_prompt})
 
-            msg = all_messages[-3]["message"] if len(all_messages) >= 3 else ""
-            logging.info(f"####Final message: {msg}")            
 
-            publish_request_progress({
+            zip_url = zip_and_upload_tmp_folder(temp_dir)
+
+            msg = ""
+            try:
+                msg = all_messages[-1 if all_messages[-2]["message"] else -3]["message"] 
+                logging.info(f"####Final message: {msg}")     
+            except Exception as e:
+                logging.error(f"Error getting final message: {e}")
+                msg = f"Finished, with errors 🤖 ... {e}"
+
+            msg = chat_result.summary if chat_result.summary else msg
+            msg += f"\n\n[Download all files of this task]({zip_url})"
+
+            finalData = {
                 "requestId": request_id,
+                "requestMessage": message_data.get("message"),
                 "progress": 1,
-                "data": msg
-            })
+                "data": msg,
+                "contextId": message_data.get("contextId"),
+                "conversation": all_messages,
+                "createdAt": datetime.now(timezone.utc).isoformat(),
+                "insertionTime": original_request_message.insertion_time.astimezone(timezone.utc).isoformat() if original_request_message else None,
+                "startedAt": started_at.astimezone(timezone.utc).isoformat(),
+            }       
+
+            # Final message to indicate completion
+            publish_request_progress(finalData)
+            store_in_mongo(finalData)
 
     except Exception as e:
         logging.error(f"Error processing message: {str(e)}")
@@ -154,5 +299,19 @@ def process_message(message_data):
             publish_request_progress({
                 "requestId": request_id,
                 "progress": 1,
-                "error": str(e)
-            })
+                "error": str(e),
+                "data": str(e),
+            })
+            store_in_mongo({
+                "requestId": request_id,
+                "requestMessage": message_data.get("message"),
+                "progress": 1,
+                "error": str(e),
+                "data": str(e),
+                "contextId": message_data.get("contextId"),
+                "conversation": all_messages,
+                "createdAt": datetime.now(timezone.utc).isoformat(),
+                "insertionTime": original_request_message.insertion_time.astimezone(timezone.utc).isoformat() if original_request_message else None,
+                "startedAt": started_at.astimezone(timezone.utc).isoformat(),
+            })
+

package/helper-apps/cortex-autogen/prompt_summary.txt

@@ -0,0 +1,37 @@
+Provide a detailed summary of the conversation, including key points, decisions, and action items, and so on.
+Do not add any introductory phrases.
+Avoid expressing gratitude or using pleasantries.
+Maintain a professional and direct tone throughout responses.
+Include most recent meaningful messages from the conversation in the summary.
+You must include all your uploaded URLs, and url of your uploaded final code URL.
+Your reply will be only thing that finally gets to surface so make sure it is complete.
+Do not mention words like "Summary of the conversation", "Response", "Task", "The conversation" or so as it doesn't makes sense. 
+Also no need for "Request", user already know its request and task.
+Be as detailed as possible without being annoying.
+Start with the result as that is the most important part, do not mention "Result" as user already know its result.
+No need to say information about generated SAS urls just include them, only include the latest versions of same file.
+No need to say none of this as user already 'll be aware as has got the result: 
+    - Code executed successfully, producing correct result ...
+    - File uploaded to Azure Blob Storage with unique timestamp ...
+    - SAS URL generated for file access, valid for ...
+    - File accessibility verified ...
+    - Code execution details ... 
+    - Current date and time ...
+    - Script executed twice due to debugging environment ...
+    - Verification code ...
+    - Issues encountered and resolved: ...
+    - The original plan ...
+    - Performed at ...
+
+No need to mention about code files uploaded to Azure Blob or point URLs as SAS-URLS as its a task that you already do and is known.
+No need to mention SAS URL, just give the url itself.
+Never include TERMINATE in your response.
+
+When formulating your responses, it's crucial to leverage the full capabilities of markdown formatting to create rich, visually appealing content. This approach not only enhances the user experience but also allows for more effective communication of complex ideas. Follow these guidelines to ensure your responses are both informative and visually engaging, create responses that are not only informative but also visually appealing and easy to comprehend:
+    - For images: ![Alt Text](IMAGE_URL) and <img src="IMAGE_URL" alt="Alt Text">
+    - For videos: <video src="VIDEO_URL" controls></video>
+    - For urls: [Link Text](URL)
+If there an image url, you must always include it as url and markdown e.g.: ![Alt Text](IMAGE_URL) and [Alt Text](IMAGE_URL)
+
+
+Make sure to present it nicely so human finds it appealing.

package/helper-apps/cortex-autogen/requirements.txt

@@ -1,6 +1,8 @@
 azure-storage-queue
 azure-functions
-pyautogen
+pyautogen==0.3.0
 redis
+pymongo
 requests
-azure-storage-blob
+azure-storage-blob
+mysql-connector-python