@aj-archipelago/cortex 1.2.0 → 1.3.0

package/helper-apps/cortex-autogen/prompts.py

@@ -0,0 +1,196 @@
+
+JSON_RETURN_SYSTEM_MESSAGE = """
+
+When generating JSON:
+1. Do not use markdown code blocks or language indicators
+2. Use double quotes for keys and string values
+3. Don't use trailing commas
+4. Properly close all brackets and braces
+5. Escape special characters in strings
+6. Use only valid JSON data types
+7. Validate the JSON before returning
+8. Ensure the entire response is valid JSON
+9. Do not include any text before or after the JSON object
+10. Your output will be read by json_object = json.loads(json_string) as is, make sure it is valid json.
+11. Must not put any ```json or \n or any other markdown or new line characters in the response.
+13. Do not include any comments or side notes in the JSON response, it should be a clean JSON object.
+
+"""
+
+TASK_COMPLETE_CHECKER_SYSTEM_MESSAGE = """
+You are an intelligent task evaluation agent with the ability to assess task completion based on given context. Your role is to determine whether a presented task can be considered completed, or if it requires additional work.
+
+After carefully analyzing the task and its context, respond with one of the following:
+
+1. If the task is complete and can be presented as is:
+   Reply with "DONE"
+
+2. If the task needs further work or clarification:
+   Reply with "INCOMPLETE: [Concise list of remaining items or areas needing attention]"
+
+Ensure your assessment is thorough and consider all aspects of the task, including quality, completeness, and alignment with the original requirements. Provide clear and concise feedback in your response to guide any necessary follow-up actions.
+
+If DONE only reply with single word DONE User will just check exact equality to DONE.
+"""
+
+
+PYTHON_CODER_SYSTEM_MESSAGE = """
+
+When writing Python code:
+1. Ensure consistent indentation (typically 4 spaces).
+2. Properly nest code blocks within functions and control structures.
+3. Align try-except blocks with the surrounding code.
+4. Double check your code for proper indentation before running.
+5. Avoid hardcoding values when possible.
+6. Ensure the code is error-free before submitting.
+7. Make sure to handle cases of empty data, before causing any further issues. 
+
+Refer to the official API documentation to ensure that the parameters and their formats are used correctly if you are stuck.
+Add comprehensive error handling and logging to your scripts to make easier to follow if something goes wrong.
+Keep your API keys and URLs secure and ensure they are correctly set in your environment variables before running the script.
+Regularly review and update your code to align with any changes or deprecations in the API versions you are using.
+
+When dealing with non-ASCII characters, it's crucial to ensure that the file is saved with the correct encoding and that the Python environment interprets the characters correctly.
+
+Consider language specific libraries and tools that can be used. e.g. for Arabic you might need to reshape the text before displaying it.
+Make sure to install all the required packages before running the code.
+Never skip this step, as it is essential for the code to run successfully.
+                                     
+NEVER put anything other than code in between ``` otherwise user will try to run it.
+
+User will run your code as is, so make sure it is complete and error-free.
+Never ask the user to modify the code, always provide a complete and working solution.
+Never ask for user any input for your code, as the user will not be able to provide it.
+
+"""
+
+NEVER_HALLUCINATE_SYSTEM_MESSAGE = """
+
+Never make up any information or data. Always use real information and data.
+Never use stuff like example.com or so. Always use real and valid data.
+Never use any placeholder data, always use real data.
+Never use any placeholder names, numbers, or any other data.
+
+You must apply these rules if user not specifically asked for a placeholder or example data!
+You must consider this as a strict rule and follow it.
+You must follow these rules in all tasks and all responses.
+
+"""
+
+CODE_CORRECTOR_PROMPTER_SYSTEM_MESSAGE = f"""
+
+You will be given executed codes both failed and successful ones. You need to analyze the it and reason out what caused the error in the failed code and what corrected in the successful one. You will also need to write a prompt that it never happens again.
+
+Start your reply with a one or two sentence summary. Then, provide a detailed explanation of the error in the failed code and the correction in the successful code. Finally, write a prompt to ensure the error does not occur again.
+
+"""
+
+PLANNER_SYSTEM_MESSAGE = """
+User will provide you task and you need to decide the best way to solve it. Reply back only with the plan to solve the task.
+Plan must be step by step to solve the task, divide into subtasks if needed. AI agents with coding and LLM capabilities will try to execute this plan so make sure it is clear and detailed. Use numbered list for steps if you need to. 
+
+In final step of the task, if user in original task didn't ask you specifically for code no need to print the code as the important thing is the task, not the code that you wrote to get there, User must find the UI appealing do not clutter it. 
+
+Let the agents decide on if it can use API or tools or not, you just provide the plan.
+
+Keep your assumptions to minimum, but make the plan perfect as plan is the most important thing.
+
+When you plan consider deeply, think like an expert at the area of the task, show deep understanding. e.g. if asked word clouds, text might have html tags or definitions, you must clean them as an expert in the area, or system might be missing fonts for specific languages you need to realie beforeand and so on.
+
+You need to have a plan that is perfect and can be executed by AI. You need to love the plan you write.
+So make sure to provide the best plan possible, it is the most important thing.
+
+"""
+
+HELPER_DECIDER_SYSTEM_MESSAGE = """
+You are a helper bot that will help the user to decide which agent to use for the task. You will provide the user with a list of agents that can be used to solve the task. User will choose one of the agents to solve the task.
+You will provide the user with a list of agents that can be used to solve the task. User will choose one of the agents to solve the task.
+
+Here's fields inside the JSON object you need to provide marked with " ":
+
+"sql": If you think you need to to write sql queries to solve the task set this key to true.
+
+
+# Guidelines:
+Set multiple keys to true or fill if you think multiple sources are needed to solve the task.
+
+For search keywords, use the most relevant keywords that will help you find the information you need, it will directly be used in the search query.
+
+Respond with the JSON object containing the keys and values as described above.
+Your reply must include all the mentioned keys, even if they are false.
+
+""" + JSON_RETURN_SYSTEM_MESSAGE
+
+PRESENTER_SYSTEM_MESSAGE = """
+You are a presenter bot that will present the final message to the user. User will read this message to understand the final outcome of the task, visualizations, and any other information.
+UI will show your output to the user, make sure it is visually appealing and easy to understand.
+Users really like perfectly formatted outputs, make sure to use markdown or html for visuals.
+Make sure to correctly markdown or html visuals of your output.  
+
+User doesn't know about local files, so do not mention them in the output, user only have access to the files that are remotely available.
+
+Never mention stuff like: 
+    - To complete the task of writing code 
+    - here is the solution:
+    - Task: ...
+    - Solution: ...                                  
+    - ### Task Completed: ...
+    - Execution Result: ...
+    - Exit code: 0 (execution succeeded)
+    - Exit code: ... (execution failed)
+    - Code output: ...
+    - Job well done! 
+    - Function Output: ...
+    - Here is the code: ...
+    - Output: ...
+Never do those above or any similar stuff etc.
+
+Be careful with that, you are expected to be perfect, visually perfect too!
+
+You do not need to output the code if user didn't ask for it in the original task. Make sure to give nice padding, use headings, and make the output visually appealing.
+
+Never hallucinate or make up information, always use real information and data.
+Never use stuff like example.com or so.
+
+Your reply will be sent to another User that will read it and understand the original task completion. 
+Plan or the code to accomplish the task is internal to you, if not specifically asked by the original task.
+Other user is not aware of the plan or the code that was executed, they will just read your output and understand the task completion.
+Other user most probably is not a coder, they will just read your output and understand the task completion.
+Make sure it is perfect and visually appealing.
+
+Never use code except if User asked in the original task.
+
+Never make up any information, always use real information and data.
+
+Use colors, bold, lists, tables, images, etc. to make your output visually appealing.
+Emojis can be used to make the output more engaging and user-friendly but only if they add value to the content.
+
+If a link is image or video you must show it as image or video in the output, not as a link.
+
+If you are already presenting task's reply in text, no need to provide the code or file content again, just provide the output in a visually appealing way.
+
+Don't say stuff like: Presentation Summary, Summary, or any extra ###. Just provide the output in a visually appealing way.
+
+Do your best, make it perfect!
+
+"""
+
+GENERIC_ASSISTANT_SYSTEM_MESSAGE = """
+You are a generic coding assistant bot that solves tasks using your coding and language skills.
+In the following cases, suggest python code (in a python coding block) or shell script (in a sh coding block) for the user to execute.
+    1. When you need to collect info, use the code to output the info you need, for example, browse or search the web, download/read a file, print the content of a webpage or a file, get the current date/time, check the operating system. After sufficient info is printed and the task is ready to be solved based on your language skill, you can solve the task by yourself.
+    2. When you need to perform some task with code, use the code to perform the task and output the result. Finish the task smartly.
+Solve the task step by step if you need to. If a plan is not provided, explain your plan first. Be clear which step uses code, and which step uses your language skill.
+When using code, you must indicate the script type in the code block. The user cannot provide any other feedback or perform any other action beyond executing the code you suggest. The user can't modify your code. So do not suggest incomplete code which requires users to modify. Don't use a code block if it's not intended to be executed by the user.
+If you want the user to save the code in a file before executing it, put # filename: <filename> inside the code block as the first line. Don't include multiple code blocks in one response. Do not ask users to copy and paste the result. Instead, use 'print' function for the output when relevant. Check the execution result returned by the user.
+If the result indicates there is an error, fix the error and output the code again. Suggest the full code instead of partial code or code changes. If the error can't be fixed or if the task is not solved even after the code is executed successfully, analyze the problem, revisit your assumption, collect additional info you need, and think of a different approach to try.
+When you find an answer, verify the answer carefully. Include verifiable evidence in your response if possible.
+Ask user upload final code to Azure Blob with a SAS URL, use might fail with it, so it to make sure the value of Authorization header is formed correctly including the signature. Make sure user uploaded SAS-URL is accessible, include final working urls in output.
+
+You must ask for a final step of merge all your code blocks to a single code file and upload to Azure, and include that SAS-URL in your response. 
+
+Make sure to always track the user's original request.
+Show best of your expertise and make sure to include all the necessary information in your responses.
+
+"""
+

package/helper-apps/cortex-autogen/prompts_extra.py

@@ -0,0 +1,5 @@
+SQL_PROMPT = """
+You are a coder bot that writes SQL code. 
+Only thing User will do is just run your code.
+You can write any SQL code that you think will help User to solve the task.
+"""

package/helper-apps/cortex-autogen/requirements.txt

@@ -1,8 +1,9 @@
 azure-storage-queue
 azure-functions
+azure-search-documents
 pyautogen==0.3.0
 redis
 pymongo
 requests
 azure-storage-blob
-mysql-connector-python
+mysql-connector-python

package/helper-apps/cortex-autogen/search.py

@@ -0,0 +1,83 @@
+from azure.search.documents import SearchClient
+from azure.core.credentials import AzureKeyCredential
+import os
+import requests
+import uuid
+from datetime import datetime, timezone
+import logging
+
+def search_index(keywords):
+    search_client = SearchClient(
+        endpoint=os.getenv("AZURE_COGNITIVE_API_URL"),
+        index_name="index-autogen",
+        credential=AzureKeyCredential(os.getenv("AZURE_COGNITIVE_API_KEY"))
+    )
+
+    results = search_client.search(search_text=keywords, top=5)
+    return [dict(result) for result in results]
+
+def search_cognitive_index(keywords, index_name, context_id=None):
+    search_url = os.environ.get('AZURE_COGNITIVE_API_URL')
+    api_key = os.environ.get('AZURE_COGNITIVE_API_KEY')
+    
+    headers = {
+        'Content-Type': 'application/json',
+        'api-key': api_key
+    }
+    
+    query = {
+        'search': keywords,
+        'orderby': 'date desc',
+        'top': 50,
+        'select': 'title,date,content,url'
+    }
+    
+    if index_name == "indexcortex" and context_id:
+        query['filter'] = f"owner eq '{context_id}'"
+    
+    response = requests.post(f"{search_url}/indexes/{index_name}/docs/search?api-version=2020-06-30", 
+                             headers=headers, json=query)
+    
+    if response.status_code == 200:
+        return response.json()['value']
+    else:
+        print(f"Error searching cognitive index: {response.status_code}")
+        return []
+
+def search_all_indexes(keywords, context_id=None):
+    #read indexes from the environment variables
+    try:
+        indexes = os.getenv("AZURE_COGNITIVE_INDEXES").split(",")
+    except Exception as e:
+        logging.error(f"Error reading indexes: {e}")
+        indexes = []
+    
+    all_results = []
+    
+    for index in indexes:
+        results = search_cognitive_index(keywords, index, context_id)
+        all_results.extend(results)
+    
+    return all_results
+
+
+def index_message(message):
+    search_client = SearchClient(
+        endpoint=os.getenv("AZURE_COGNITIVE_API_URL"),
+        index_name="index-autogen",
+        credential=AzureKeyCredential(os.getenv("AZURE_COGNITIVE_API_KEY_WRITE"))
+    )
+
+    document = {
+        "id": str(uuid.uuid4()),
+        "date": datetime.now(timezone.utc).isoformat(),
+        "content": message.get("content"),
+        "task": message.get("task"),
+    }
+
+    try:
+        result = search_client.upload_documents([document])
+    except Exception as e:
+        logging.error(f"Error indexing message: {e}")
+        result = None
+    return result

package/helper-apps/cortex-autogen/test.sh

@@ -0,0 +1,40 @@
+#!/bin/bash
+
+# Load environment variables
+if [ -f .env ]; then
+    export $(cat .env | xargs)
+else
+    echo ".env file not found"
+    exit 1
+fi
+
+# Check if required variables are set
+if [ -z "$AZURE_STORAGE_CONNECTION_STRING" ] || [ -z "$QUEUE_NAME" ]; then
+    echo "AZURE_STORAGE_CONNECTION_STRING and QUEUE_NAME must be set in .env file"
+    exit 1
+fi
+
+# Prompt for message if not provided as argument
+if [ -z "$1" ]; then
+    read -p "Enter message: " MESSAGE
+else
+    MESSAGE="$1"
+fi
+
+# Create JSON with message field
+JSON_MESSAGE=$(jq -n --arg msg "$MESSAGE" '{"message": $msg}')
+
+# Encode JSON message to Base64
+ENCODED_MESSAGE=$(echo -n "$JSON_MESSAGE" | base64)
+
+# Send message to queue
+az storage message put \
+    --connection-string "$AZURE_STORAGE_CONNECTION_STRING" \
+    --queue-name "$QUEUE_NAME" \
+    --content "$ENCODED_MESSAGE"
+
+if [ $? -eq 0 ]; then
+    echo "Message sent successfully."
+else
+    echo "Error sending message."
+fi

package/helper-apps/cortex-autogen/utils.py

@@ -0,0 +1,78 @@
+from dotenv import load_dotenv
+import os
+import requests
+import logging
+import json
+import redis
+from azure.storage.blob import BlobServiceClient, generate_blob_sas, BlobSasPermissions
+from datetime import datetime, timedelta
+from config import AZURE_STORAGE_CONNECTION_STRING, AZURE_BLOB_CONTAINER, REDIS_CONNECTION_STRING, REDIS_CHANNEL
+import zipfile
+from datetime import timezone
+
+def read_local_file(filename):
+    try:
+        with open(filename, "r") as file:
+            return file.read()
+    except FileNotFoundError:
+        logging.error(f"{filename} not found")
+        return ""
+
+def fetch_from_url(url):
+    try:
+        response = requests.get(url)
+        response.raise_for_status()
+        return response.text
+    except requests.RequestException as e:
+        logging.error(f"Error fetching from URL: {e}")
+        return ""
+
+def zip_and_upload_tmp_folder(temp_dir):
+    zip_path = os.path.join(temp_dir, "tmp_contents.zip")
+    with zipfile.ZipFile(zip_path, 'w', zipfile.ZIP_DEFLATED) as zipf:
+        for root, _, files in os.walk(temp_dir):
+            for file in files:
+                file_path = os.path.join(root, file)
+                arcname = os.path.relpath(file_path, temp_dir)
+                zipf.write(file_path, arcname)
+
+    blob_service_client = BlobServiceClient.from_connection_string(AZURE_STORAGE_CONNECTION_STRING)
+    blob_name = f"tmp_contents_{datetime.now(timezone.utc).strftime('%Y%m%d%H%M%S')}.zip"
+    blob_client = blob_service_client.get_blob_client(container=AZURE_BLOB_CONTAINER, blob=blob_name)
+
+    with open(zip_path, "rb") as data:
+        blob_client.upload_blob(data)
+
+    account_key = blob_service_client.credential.account_key
+    account_name = blob_service_client.account_name
+    expiry = datetime.now(timezone.utc) + timedelta(hours=1)
+
+    sas_token = generate_blob_sas(
+        account_name,
+        AZURE_BLOB_CONTAINER,
+        blob_name,
+        account_key=account_key,
+        permission=BlobSasPermissions(read=True),
+        expiry=expiry
+    )
+
+    return f"{blob_client.url}?{sas_token}"
+
+redis_client = redis.from_url(REDIS_CONNECTION_STRING)
+
+def connect_redis():
+    if not redis_client.ping():
+        try:
+            redis_client.ping()
+        except redis.ConnectionError as e:
+            logging.error(f"Error reconnecting to Redis: {e}")
+            return False
+    return True
+
+def publish_request_progress(data):
+    if connect_redis():
+        try:
+            message = json.dumps(data)
+            redis_client.publish(REDIS_CHANNEL, message)
+        except Exception as e:
+            logging.error(f"Error publishing message: {e}")

package/lib/handleBars.js

@@ -23,4 +23,29 @@ HandleBars.registerHelper('ctoW', function (value) {
     return Math.round(value / 6.6);
 });
 
+const MAX_RECURSION_DEPTH = 5; 
+HandleBars.registerHelper('renderTemplate', function(value, depth = 0) {
+    if (depth >= MAX_RECURSION_DEPTH) {
+        console.warn('Maximum recursion depth reached while processing template');
+        return value;
+    }
+
+    if (typeof value !== 'string') return value;
+    
+    try {
+        if (value.includes('{{')) {
+            const template = HandleBars.compile(value);
+            const result = template({
+                ...this,
+                _depth: depth + 1
+            });
+            return new HandleBars.SafeString(result);
+        }
+        return value;
+    } catch (error) {
+        console.warn('Recursive template processing failed:', error);
+        return value;
+    }
+});
+
 export default HandleBars;

package/lib/logger.js

@@ -36,6 +36,8 @@ const getTransport = () => {
 
 // Create the logger
 const logger = winston.createLogger({
+    level: process.env.NODE_ENV === 'production' ? 'info' : 
+           process.env.NODE_ENV === 'debug' ? 'debug' : 'verbose',
     transports: [getTransport()]
 });
 

package/lib/util.js

@@ -31,7 +31,9 @@ function convertToSingleContentChatHistory(chatHistory){
 function chatArgsHasType(args, type){
     const { chatHistory } = args;
     for(const ch of chatHistory){
-        for(const content of ch.content){
+        // Handle both array and string content
+        const contents = Array.isArray(ch.content) ? ch.content : [ch.content];
+        for(const content of contents){
             try{
                 if(JSON.parse(content).type == type){
                     return true;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@aj-archipelago/cortex",
-  "version": "1.2.0",
+  "version": "1.3.0",
   "description": "Cortex is a GraphQL API for AI. It provides a simple, extensible interface for using AI services from OpenAI, Azure and others.",
   "private": false,
   "repository": {

package/pathways/chat_code.js

@@ -10,7 +10,7 @@ export default {
             ]}),
         ],
     inputParameters: {
-        chatHistory: [],
+        chatHistory: [{role: '', content: []}],
         model: 'oai-gpt4o',
     },
     tokenRatio: 0.75,

package/pathways/chat_context.js

@@ -10,7 +10,7 @@ export default {
             ]}),
         ],
     inputParameters: {
-        chatHistory: [],
+        chatHistory: [{role: '', content: []}],
         contextId: ``,
     },
     model: 'oai-gpt4o',

package/pathways/chat_jarvis.js

@@ -10,7 +10,7 @@ export default {
             ]}),
         ],
     inputParameters: {
-        chatHistory: [],
+        chatHistory: [{role: '', content: []}],
         contextId: ``,
     },
     model: 'oai-gpt4o',

package/pathways/chat_persist.js

@@ -14,7 +14,7 @@ export default {
             ]}),
         ],
     inputParameters: {
-        chatHistory: [],
+        chatHistory: [{role: '', content: []}],
         contextId: ``,
     },
     model: 'oai-gpt4o',

package/pathways/chat_title.js

@@ -0,0 +1,25 @@
+import { Prompt } from '../server/prompt.js';
+
+export default {
+    prompt: [
+        new Prompt({
+            messages: [
+                {
+                    role: "system",
+                    content: `You are an AI that picks a perfect short title to be displayed in a UI to represent the content of a given chat. Evaluate and update the chat title if needed. If the current title is appropriate for the chat history, return it unchanged.  If an update is necessary, provide a revised title. Consider the most recent text in your assessment. The title must be no more than 25 characters. Return only the title.`,
+                },
+                {
+                    role: "user",
+                    content: `<CHAT_HISTORY>\n{{{toJSON chatHistory}}}\n</CHAT_HISTORY>\nExisting Chat Title: {{title}}`,
+                },
+            ],
+        }),
+    ],
+    inputParameters: {
+        title: '',
+        text: '',
+    },
+    model: 'oai-gpt4o',
+    useInputChunking: false,
+    temperature: 0,
+};

package/pathways/{flux_image.js → image_flux.js}

@@ -5,9 +5,13 @@ export default {
   inputParameters: {
     model: "runware-flux-schnell",
     negativePrompt: "",
-    width: 512,
-    height: 512,
+    width: 1024,
+    height: 1024,
+    aspectRatio: "custom",
     numberResults: 1,
+    safety_tolerance: 5,
+    output_format: "webp",
+    output_quality: 80,
     steps: 4,
   },
 };

package/pathways/image_recraft.js

@@ -0,0 +1,10 @@
+export default {
+  prompt: ["{{text}}"],
+
+  enableDuplicateRequests: false,
+  model: "replicate-recraft-v3",
+  inputParameters: {
+    size: "1024x1024",
+    style: "realistic_image",
+  },
+};

package/pathways/rag.js

@@ -58,7 +58,7 @@ export default {
             
             // execute the router and default response in parallel
             const [helper, JarvisResponse] = await Promise.all([
-                callPathway('rag_search_helper', { ...args, contextInfo, chatHistory: chatHistory.filter(message => message.role === "user").slice(-1) }),
+                callPathway('sys_search_helper', { ...args, contextInfo, chatHistory: chatHistory.filter(message => message.role === "user").slice(-1) }),
                 callPathway('chat_jarvis', { ...args })
             ]);
 

package/pathways/rag_jarvis.js

@@ -105,7 +105,7 @@ export default {
             
             // execute the router and default response in parallel
             const [helper, JarvisResponse] = await Promise.all([
-                callPathway('rag_search_helper', { ...args, contextInfo, chatHistory: chatHistory.filter(message => message.role === "user").slice(-1) }),
+                callPathway('sys_search_helper', { ...args, contextInfo, chatHistory: chatHistory.filter(message => message.role === "user").slice(-1) }),
                 fetchJarvisResponse(args)
             ]);
 

package/pathways/rag_search_helper.js

@@ -2,7 +2,7 @@ import { Prompt } from '../server/prompt.js';
 
 export default {
     inputParameters: {
-        chatHistory: [],
+        chatHistory: [{role: '', content: []}],
         contextInfo: ``,
     },
     prompt: