@aj-archipelago/cortex 1.3.61 → 1.3.63

package/.github/workflows/cortex-file-handler-test.yml

@@ -0,0 +1,61 @@
+name: Cortex File Handler Tests
+
+on:
+  push:
+    branches: [ main, dev ]
+    paths:
+      - 'helper-apps/cortex-file-handler/**'
+      - '.github/workflows/cortex-file-handler-test.yml'
+  pull_request:
+    branches: [ main, dev ]
+    paths:
+      - 'helper-apps/cortex-file-handler/**'
+      - '.github/workflows/cortex-file-handler-test.yml'
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    
+    
+    defaults:
+      run:
+        working-directory: helper-apps/cortex-file-handler
+    
+    steps:
+    - name: Checkout code
+      uses: actions/checkout@v4
+    
+    - name: Setup Node.js 20.x
+      uses: actions/setup-node@v4
+      with:
+        node-version: '20.x'
+        cache: 'npm'
+        cache-dependency-path: helper-apps/cortex-file-handler/package-lock.json
+    
+    - name: Install system dependencies
+      run: |
+        sudo apt-get update
+        sudo apt-get install -y ffmpeg
+    
+    - name: Install dependencies
+      run: npm ci
+    
+    - name: Install Azurite
+      run: npm install -g azurite
+    
+    - name: Setup GCS test environment
+      run: cp .env.test.gcs.ci .env.test.gcs
+    
+    - name: Setup Azure test environment
+      run: cp .env.test.azure.ci .env.test.azure
+    
+    - name: Run GCS tests
+      run: npm run test:gcs
+    
+    - name: Run Azure tests
+      run: npm run test:azure
+    
+    - name: Run tests
+      run: npm test
+      env:
+        NODE_ENV: test

package/README.md

@@ -1,5 +1,6 @@
 # Cortex
-Cortex simplifies and accelerates the process of creating applications that harness the power of modern AI models like GPT-4o (chatGPT), o1, o3-mini, Gemini, the Claude series, Flux, Grok and more by poviding a structured interface (GraphQL or REST) to a powerful prompt execution environment. This enables complex augmented prompting and abstracts away most of the complexity of managing model connections like chunking input, rate limiting, formatting output, caching, and handling errors.
+Cortex simplifies and accelerates the process of creating applications that harness the power of modern AI models like GPT-5 (chatGPT), o4, Gemini, the Claude series, Flux, Grok and more by poviding a structured interface (GraphQL or REST) to a powerful prompt execution environment. This enables complex augmented prompting and abstracts away most of the complexity of managing model connections like chunking input, rate limiting, formatting output, caching, and handling errors.
+
 ## Why build Cortex?
 Modern AI models are transformational, but a number of complexities emerge when developers start using them to deliver application-ready functions. Most models require precisely formatted, carefully engineered and sequenced prompts to produce consistent results, and the responses are typically largely unstructured text without validation or formatting. Additionally, these models are evolving rapidly, are typically costly and slow to query and implement hard request size and rate restrictions that need to be carefully navigated for optimum throughput. Cortex offers a solution to these problems and provides a simple and extensible package for interacting with NL AI models.
 
@@ -18,6 +19,7 @@ Just about anything! It's kind of an LLM swiss army knife.  Here are some ideas:
 * Simple architecture to build custom functional endpoints (called `pathways`), that implement common NL AI tasks. Default pathways include chat, summarization, translation, paraphrasing, completion, spelling and grammar correction, entity extraction, sentiment analysis, and bias analysis.
 * Extensive model support with built-in integrations for:
   - OpenAI models:
+    - GPT-5 (all flavors and router)
     - GPT-4.1 (+mini, +nano)
     - GPT-4 Omni (GPT-4o)
     - O3 and O4-mini (Advanced reasoning models)
@@ -28,10 +30,15 @@ Just about anything! It's kind of an LLM swiss army knife.  Here are some ideas:
     - Gemini 2.0 Flash
     - Earlier Google models (Gemini 1.5 series)
   - Anthropic models:
-    - Claude 3.7 Sonnet
-    - Claude 3.5 Sonnet
-    - Claude 3.5 Haiku
-    - Claude 3 Series
+  - Claude 4 Sonnet (Vertex)
+  - Claude 4.1 Opus (Vertex)
+  - Claude 3.7 Sonnet
+  - Claude 3.5 Sonnet
+  - Claude 3.5 Haiku
+  - Claude 3 Series
+  - Grok (XAI) models:
+    - Grok 3 and Grok 4 series (including fast-reasoning and code-fast variants)
+    - Multimodal chat with vision, streaming, and tool calling
   - Ollama support
   - Azure OpenAI support
   - Custom model implementations
@@ -419,6 +426,11 @@ Each pathway can define the following properties (with defaults from basePathway
 - `temperature`: Model temperature setting (0.0 to 1.0). Default: 0.9
 - `json`: Require valid JSON response from model. Default: false
 - `manageTokenLength`: Manage input token length for model. Default: true
+  
+#### Dynamic model override
+
+- `model`: In many cases, specifying the model as an input parameter will tell the pathway which model to use when setting up the pathway for execution.
+- `modelOverride`: In some cases, you need even more dynamic model selection. At runtime, a pathway can optionally specify `modelOverride` in request args to switch the model used for execution without restarting the server. Cortex will attempt a hot swap and continue execution; errors are logged gracefully if the model is invalid.
 
 ## Core (Default) Pathways
 
@@ -528,6 +540,7 @@ Models are configured in the `models` section of the config. Each model can have
 - `GEMINI-1.5-CHAT`: For Gemini 1.5 Pro chat models
 - `GEMINI-1.5-VISION`: For Gemini vision models (including 2.0 Flash experimental)
 - `CLAUDE-3-VERTEX`: For Claude-3 and 3.5 models (Haiku, Opus, Sonnet)
+- `GROK-VISION`: For XAI Grok models (Grok-3, Grok-4, fast-reasoning, code-fast) with multimodal/vision and reasoning
 - `AZURE-TRANSLATE`: For Azure translation services
 
 Each model configuration can include:
@@ -606,6 +619,18 @@ To enable Ollama support, add the following to your configuration:
 }
 ```
 
+#### Tool Calling and Structured Responses
+
+When using the OpenAI-compatible REST endpoints, Cortex supports vendor-agnostic tool calling with OpenAI-style `tool_calls` deltas in streaming mode. Pathway responses now include a structured `resultData` field (also exposed via GraphQL) that may contain:
+
+- `toolCalls` and/or `functionCall` objects
+- vendor-specific metadata (e.g., search citations)
+- usage details
+
+Notes:
+- `tool_choice` accepts either a string (e.g., `"auto"`, `"required"`) or an object (`{ type: 'function', function: 'name' }`); Cortex normalizes this across vendors (OpenAI, Claude via Vertex, Gemini, Grok).
+- Arrays for `[String]` inputs are passed directly through REST conversion.
+
 You can then use any Ollama model through the standard OpenAI-compatible endpoints:
 
 ```bash
@@ -846,8 +871,7 @@ Cortex includes a powerful Entity System that allows you to build autonomous age
 ### Overview
 
 The Entity System is built around two core pathways:
-- `sys_entity_start.js`: The entry point for entity interactions, handling initial routing and tool selection
-- `sys_entity_continue.js`: Manages callback execution in synchronous mode
+- `sys_entity_agent.js`: The entry point for entity interactions, handling initial routing and tool selection
 
 ### Key Features
 

package/config/default.example.json

@@ -159,6 +159,21 @@
             ],
             "requestsPerSecond": 10,
             "maxTokenLength": 1024
+        },
+        "xai-grok-3": {
+            "type": "GROK-VISION",
+            "url": "https://api.x.ai/v1/chat/completions",
+            "headers": {
+                "Authorization": "Bearer {{XAI_API_KEY}}",
+                "Content-Type": "application/json"
+            },
+            "params": {
+                "model": "grok-3-latest"
+            },
+            "requestsPerSecond": 10,
+            "maxTokenLength": 131072,
+            "maxReturnTokens": 4096,
+            "supportsStreaming": true
         }
     },
     "enableCache": false,

package/config.js

@@ -59,7 +59,7 @@ var config = convict({
     },
     defaultModelName: {
         format: String,
-        default: null,
+        default: 'oai-gpt4o',
         env: 'DEFAULT_MODEL_NAME'
     },
     defaultEntityName: {
@@ -94,12 +94,12 @@ var config = convict({
     },
     claudeVertexUrl: {
         format: String,
-        default: 'https://region.googleapis.com/v1/projects/projectid/locations/location/publishers/anthropic/models/claude-3-5-sonnet@20240620',
+        default: 'https://region.googleapis.com/v1/projects/projectid/locations/location/publishers/anthropic/models/claude-4-sonnet@20250722',
         env: 'CLAUDE_VERTEX_URL'
     },
     geminiFlashUrl: {
         format: String,
-        default: 'https://region.googleapis.com/v1/projects/projectid/locations/location/publishers/google/models/gemini-2.0-flash-001',
+        default: 'https://region.googleapis.com/v1/projects/projectid/locations/location/publishers/google/models/gemini-2.5-flash',
         env: 'GEMINI_FLASH_URL'
     },
     entityConfig: {
@@ -295,6 +295,21 @@ var config = convict({
                 "maxReturnTokens": 100000,
                 "supportsStreaming": false
             },
+            "oai-o3": {
+                "type": "OPENAI-REASONING",
+                "url": "https://api.openai.com/v1/chat/completions",
+                "headers": {
+                    "Authorization": "Bearer {{OPENAI_API_KEY}}",
+                    "Content-Type": "application/json"
+                },
+                "params": {
+                    "model": "o3"
+                },
+                "requestsPerSecond": 10,
+                "maxTokenLength": 200000,
+                "maxReturnTokens": 100000,
+                "supportsStreaming": true
+            },
             "oai-o3-mini": {
                 "type": "OPENAI-REASONING",
                 "url": "https://api.openai.com/v1/chat/completions",
@@ -308,7 +323,7 @@ var config = convict({
                 "requestsPerSecond": 10,
                 "maxTokenLength": 200000,
                 "maxReturnTokens": 100000,
-                "supportsStreaming": false
+                "supportsStreaming": true
             },
             "azure-bing": {
                 "type": "AZURE-BING",
@@ -320,6 +335,15 @@ var config = convict({
                 "requestsPerSecond": 10,
                 "maxTokenLength": 200000
             },
+            "google-cse": {
+                "type": "GOOGLE-CSE",
+                "url": "https://www.googleapis.com/customsearch/v1",
+                "headers": {
+                    "Content-Type": "application/json"
+                },
+                "requestsPerSecond": 10,
+                "maxTokenLength": 200000
+            },
             "runware-flux-schnell": {
                 "type": "RUNWARE-AI",
                 "url": "https://api.runware.ai/v1",
@@ -449,7 +473,7 @@ var config = convict({
                 "maxReturnTokens": 4096,
                 "supportsStreaming": true
             },
-            "claude-35-sonnet-vertex": {
+            "claude-37-sonnet-vertex": {
                 "type": "CLAUDE-3-VERTEX",
                 "url": "{{claudeVertexUrl}}",
                 "headers": {
@@ -461,15 +485,102 @@ var config = convict({
                 "maxImageSize": 5242880,
                 "supportsStreaming": true
             },
-            "gemini-flash-20-vision": {
-                "type": "GEMINI-1.5-VISION",
-                "url": "{{geminiFlashUrl}}",
+            "claude-4-sonnet-vertex": {
+                "type": "CLAUDE-3-VERTEX",
+                "url": "{{claudeVertexUrl}}",
                 "headers": {
                     "Content-Type": "application/json"
                 },
                 "requestsPerSecond": 10,
                 "maxTokenLength": 200000,
                 "maxReturnTokens": 4096,
+                "maxImageSize": 5242880,
+                "supportsStreaming": true
+            },
+            "gemini-flash-25-vision": {
+                "type": "GEMINI-1.5-VISION",
+                "url": "{{geminiFlashUrl}}",
+                "headers": {
+                    "Content-Type": "application/json"
+                },
+                "requestsPerSecond": 10,
+                "maxTokenLength": 1048576,
+                "maxReturnTokens": 65535,
+                "supportsStreaming": true
+            },
+            "xai-grok-3": {
+                "type": "GROK-VISION",
+                "url": "https://api.x.ai/v1/chat/completions",
+                "headers": {
+                    "Authorization": "Bearer {{XAI_API_KEY}}",
+                    "Content-Type": "application/json"
+                },
+                "params": {
+                    "model": "grok-3-latest"
+                },
+                "requestsPerSecond": 10,
+                "maxTokenLength": 131072,
+                "maxReturnTokens": 32000,
+                "supportsStreaming": true
+            },
+            "xai-grok-4": {
+                "type": "GROK-VISION",
+                "url": "https://api.x.ai/v1/chat/completions",
+                "headers": {
+                    "Authorization": "Bearer {{XAI_API_KEY}}",
+                    "Content-Type": "application/json"
+                },
+                "params": {
+                    "model": "grok-4-0709"
+                },
+                "requestsPerSecond": 10,
+                "maxTokenLength": 256000,
+                "maxReturnTokens": 128000,
+                "supportsStreaming": true
+            },
+            "xai-grok-code-fast-1": {
+                "type": "GROK-VISION",
+                "url": "https://api.x.ai/v1/chat/completions",
+                "headers": {
+                    "Authorization": "Bearer {{XAI_API_KEY}}",
+                    "Content-Type": "application/json"
+                },
+                "params": {
+                    "model": "grok-code-fast-1"
+                },
+                "requestsPerSecond": 10,
+                "maxTokenLength": 2000000,
+                "maxReturnTokens": 128000,
+                "supportsStreaming": true
+            },
+            "xai-grok-4-fast-reasoning": {
+                "type": "GROK-VISION",
+                "url": "https://api.x.ai/v1/chat/completions",
+                "headers": {
+                    "Authorization": "Bearer {{XAI_API_KEY}}",
+                    "Content-Type": "application/json"
+                },
+                "params": {
+                    "model": "grok-4-fast-reasoning"
+                },
+                "requestsPerSecond": 10,
+                "maxTokenLength": 2000000,
+                "maxReturnTokens": 128000,
+                "supportsStreaming": true
+            },
+            "xai-grok-4-fast-non-reasoning": {
+                "type": "GROK-VISION",
+                "url": "https://api.x.ai/v1/chat/completions",
+                "headers": {
+                    "Authorization": "Bearer {{XAI_API_KEY}}",
+                    "Content-Type": "application/json"
+                },
+                "params": {
+                    "model": "grok-4-fast-non-reasoning"
+                },
+                "requestsPerSecond": 10,
+                "maxTokenLength": 256000,
+                "maxReturnTokens": 128000,
                 "supportsStreaming": true
             },
             "apptek-translate": {
@@ -483,6 +594,21 @@ var config = convict({
                 "requestsPerSecond": 10,
                 "maxTokenLength": 128000
             },
+            "azure-bing-agent": {
+                "type": "AZURE-FOUNDRY-AGENTS",
+                "url": "{{azureFoundryAgentUrl}}",
+                "headers": {
+                    "Content-Type": "application/json"
+                },
+                "params": {
+                    "api-version": "2025-05-01",
+                    "assistant_id": "{{azureFoundryAgentId}}"
+                },
+                "requestsPerSecond": 10,
+                "maxTokenLength": 32768,
+                "maxReturnTokens": 4096,
+                "supportsStreaming": false
+            }
         },
         env: 'CORTEX_MODELS'
     },
@@ -592,6 +718,21 @@ var config = convict({
         format: String,
         default: null,
         env: 'APPTEK_API_ENDPOINT'
+    },
+    azureFoundryAgentUrl: {
+        format: String,
+        default: null,
+        env: 'AZURE_FOUNDRY_AGENT_URL'
+    },
+    azureFoundryAgentId: {
+        format: String,
+        default: null,
+        env: 'AZURE_FOUNDRY_AGENT_ID'
+    },
+    azureFoundryBingSearchConnectionId: {
+        format: String,
+        default: null,
+        env: 'AZURE_FOUNDRY_BING_SEARCH_CONNECTION_ID'
     }
 });
 
@@ -716,10 +857,15 @@ const buildPathways = async (config) => {
                     Object.assign(pathways, subPathways);
                 } else if (file.name.endsWith('.js')) {
                     // Load individual pathway file
-                    const pathwayURL = pathToFileURL(fullPath).toString();
-                    const pathway = await import(pathwayURL).then(module => module.default || module);
-                    const pathwayName = path.basename(file.name, '.js');
-                    pathways[pathwayName] = pathway;
+                    try {
+                        const pathwayURL = pathToFileURL(fullPath).toString();
+                        const pathway = await import(pathwayURL).then(module => module.default || module);
+                        const pathwayName = path.basename(file.name, '.js');
+                        pathways[pathwayName] = pathway;
+                    } catch (pathwayError) {
+                        logger.error(`Error loading pathway file ${fullPath}: ${pathwayError.message}`);
+                        throw pathwayError; // Re-throw to be caught by outer catch block
+                    }
                 }
             }
         } catch (error) {

package/helper-apps/cortex-autogen2/DigiCertGlobalRootCA.crt.pem

@@ -0,0 +1,22 @@
+-----BEGIN CERTIFICATE-----
+MIIDrzCCApegAwIBAgIQCDvgVpBCRrGhdWrJWZHHSjANBgkqhkiG9w0BAQUFADBh
+MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3
+d3cuZGlnaWNlcnQuY29tMSAwHgYDVQQDExdEaWdpQ2VydCBHbG9iYWwgUm9vdCBD
+QTAeFw0wNjExMTAwMDAwMDBaFw0zMTExMTAwMDAwMDBaMGExCzAJBgNVBAYTAlVT
+MRUwEwYDVQQKEwxEaWdpQ2VydCBJbmMxGTAXBgNVBAsTEHd3dy5kaWdpY2VydC5j
+b20xIDAeBgNVBAMTF0RpZ2lDZXJ0IEdsb2JhbCBSb290IENBMIIBIjANBgkqhkiG
+9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4jvhEXLeqKTTo1eqUKKPC3eQyaKl7hLOllsB
+CSDMAZOnTjC3U/dDxGkAV53ijSLdhwZAAIEJzs4bg7/fzTtxRuLWZscFs3YnFo97
+nh6Vfe63SKMI2tavegw5BmV/Sl0fvBf4q77uKNd0f3p4mVmFaG5cIzJLv07A6Fpt
+43C/dxC//AH2hdmoRBBYMql1GNXRor5H4idq9Joz+EkIYIvUX7Q6hL+hqkpMfT7P
+T19sdl6gSzeRntwi5m3OFBqOasv+zbMUZBfHWymeMr/y7vrTC0LUq7dBMtoM1O/4
+gdW7jVg/tRvoSSiicNoxBN33shbyTApOB6jtSj1etX+jkMOvJwIDAQABo2MwYTAO
+BgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUA95QNVbR
+TLtm8KPiGxvDl7I90VUwHwYDVR0jBBgwFoAUA95QNVbRTLtm8KPiGxvDl7I90VUw
+DQYJKoZIhvcNAQEFBQADggEBAMucN6pIExIK+t1EnE9SsPTfrgT1eXkIoyQY/Esr
+hMAtudXH/vTBH1jLuG2cenTnmCmrEbXjcKChzUyImZOMkXDiqw8cvpOp/2PV5Adg
+06O/nVsJ8dWO41P0jmP6P6fbtGbfYmbW0W5BjfIttep3Sp+dWOIrWcBAI+0tKIJF
+PnlUkiaY4IBIqDfv8NZ5YBberOgOzW6sRBc4L0na4UU+Krk2U886UAb3LujEV0ls
+YSEY1QSteDwsOoBrp+uvFRTp2InBuThs4pFsiv9kuXclVzDAGySj4dzp30d8tbQk
+CAUw7C29C79Fv1C5qfPrmAESrciIxpg0X40KPMbp1ZWVbd4=
+-----END CERTIFICATE-----

package/helper-apps/cortex-autogen2/Dockerfile

@@ -0,0 +1,31 @@
+# Use the official Azure Functions Python base image
+FROM mcr.microsoft.com/azure-functions/python:4-python3.11
+
+# Set the working directory
+WORKDIR /home/site/wwwroot
+
+# Install system dependencies
+RUN apt-get update && apt-get install -y \
+    build-essential \
+    curl \
+    git \
+    libgirepository1.0-dev \
+    pkg-config \
+    libfreetype6-dev \
+    libpng-dev \
+    fontconfig \
+    && rm -rf /var/lib/apt/lists/*
+RUN curl https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.gpg \
+    && mv microsoft.gpg /etc/apt/trusted.gpg.d/microsoft.gpg \
+    && sh -c 'echo "deb [arch=amd64,arm64,armhf] https://packages.microsoft.com/repos/microsoft-debian-bullseye-prod bullseye main" > /etc/apt/sources.list.d/azure-cli.list' \
+    && apt-get update \
+    && apt-get install -y azure-functions-core-tools
+
+# Copy requirements and install Python dependencies
+COPY requirements.txt .
+RUN pip install --no-cache-dir -r requirements.txt
+
+# Copy application code
+COPY . .
+CMD ["func", "start", "--port", "80", "--verbose"]
+

package/helper-apps/cortex-autogen2/Dockerfile.worker

@@ -0,0 +1,41 @@
+# Use an official Python runtime as a parent image
+FROM python:3.11-slim
+
+# Set the working directory in the container
+WORKDIR /app
+
+# Install system dependencies that might be needed
+RUN apt-get update && apt-get install -y \
+    gcc \
+    curl \
+    && rm -rf /var/lib/apt/lists/*
+
+# Install poetry
+RUN pip install poetry
+
+# Configure poetry: Don't create virtual env (we're already in container)
+RUN poetry config virtualenvs.create false
+
+# Copy project descriptor files first (for better Docker layer caching)
+COPY pyproject.toml poetry.lock* ./
+
+# Create the coding directory inside the container
+RUN mkdir -p /app/coding
+
+# Create a minimal README.md if it's needed for build context
+RUN touch README.md
+
+# Copy the application's code before installing dependencies
+COPY . /app/src/cortex_autogen2
+
+# Install dependencies (including the current project)
+RUN poetry install --without dev --no-ansi --no-interaction
+
+# Install Playwright browser binaries
+RUN playwright install
+
+# Ensure Python can import our source package
+ENV PYTHONPATH="/app/src:/app/src/cortex_autogen2"
+
+# The command to run the application
+CMD ["python", "-m", "cortex_autogen2.main"]