@aj-archipelago/cortex 1.3.50 → 1.3.52

package/helper-apps/cortex-browser/requirements.txt

@@ -4,14 +4,13 @@ certifi==2025.4.26
 charset-normalizer==3.4.2
 courlan==1.3.2
 dateparser==1.2.1
-greenlet==3.2.1
+greenlet==3.0.3
 htmldate==1.9.3
 jusText==3.0.2
 lxml==5.4.0
 lxml_html_clean==0.4.2
 MarkupSafe==3.0.2
-playwright==1.52.0
-pyee==13.0.0
+pyee==11.1.0
 python-dateutil==2.9.0.post0
 pytz==2025.2
 regex==2024.11.6
@@ -20,5 +19,6 @@ tld==0.13
 trafilatura==2.0.0
 typing_extensions==4.13.2
 tzlocal==5.3.1
-urllib3==2.4.0
 Werkzeug==3.1.3
+playwright==1.45.0
+aiohttp

package/helper-apps/cortex-file-handler/{.env.test.azure → .env.test.azure.sample}

@@ -3,4 +3,5 @@ REDIS_CONNECTION_STRING=redis://default:redispw@localhost:32768
 AZURE_STORAGE_CONNECTION_STRING=UseDevelopmentStorage=true
 AZURE_STORAGE_CONTAINER_NAME=test-container
 NODE_ENV=test
-PORT=7072  # Different port for testing 
+PORT=7072  # Different port for testing 
+MARKITDOWN_CONVERT_URL= #cortex-markitdown url

package/helper-apps/cortex-file-handler/{.env.test.gcs → .env.test.gcs.sample}

@@ -6,4 +6,5 @@ GCS_BUCKETNAME=cortextempfiles
 AZURE_STORAGE_CONNECTION_STRING=UseDevelopmentStorage=true
 AZURE_STORAGE_CONTAINER_NAME=test-container
 NODE_ENV=test
-PORT=7072  # Different port for testing 
+PORT=7072  # Different port for testing 
+MARKITDOWN_CONVERT_URL= #cortex-markitdown url

package/helper-apps/cortex-file-handler/{.env.test → .env.test.sample}

@@ -4,4 +4,5 @@ REDIS_CONNECTION_STRING=redis://default:redispw@localhost:32768
 AZURE_STORAGE_CONTAINER_NAME=test-container
 #GCP_SERVICE_ACCOUNT_KEY={"type":"service_account","project_id":"test-project"}
 NODE_ENV=test
-PORT=7072  # Different port for testing 
+PORT=7072  # Different port for testing 
+MARKITDOWN_CONVERT_URL= #cortex-markitdown url

package/helper-apps/cortex-file-handler/Dockerfile

@@ -16,4 +16,4 @@ EXPOSE 7071
 
 # RUN npm run build
 
-CMD [ "node", "start.js" ]
+CMD [ "npm", "start" ]

package/helper-apps/cortex-file-handler/INTERFACE.md

@@ -0,0 +1,178 @@
+# Cortex File Handler Interface Documentation
+
+## Overview
+The Cortex File Handler is a service that processes files through various operations including uploading, downloading, chunking, and document processing. It supports multiple storage backends (Azure Blob Storage, Google Cloud Storage, and Local File System).
+
+## Request Methods
+
+### POST
+- **Purpose**: Upload a file
+- **Content-Type**: `multipart/form-data`
+- **Parameters**:
+  - `hash` (optional): Unique identifier for the file
+  - `requestId` (required): Unique identifier for the request
+  - File content must be included in the form data
+- **Behavior**: 
+  - Uploads file to primary storage (Azure or Local)
+  - If GCS is configured, also uploads to GCS
+  - If hash is provided, stores file metadata in Redis
+  - Returns upload result with file URLs
+- **Response**: Object containing:
+  - `url`: Primary storage URL
+  - `gcs`: GCS URL (if GCS is configured)
+  - `hash`: Hash value (if provided)
+  - `message`: Success message
+  - `filename`: Original filename
+- **Note**: The `save` parameter is not supported in POST requests. To convert and save a document as text, use GET with the `save` parameter.
+
+### GET
+- **Purpose**: Process or retrieve files
+- **Parameters** (can be in query string or request body):
+  - `uri` (required if not using fetch/load/restore): URL of the file to process
+    - Requires `requestId` parameter
+    - No Redis caching
+    - Direct processing based on file type
+  - `requestId` (required with `uri`): Unique identifier for the request
+  - `save` (optional): If true, saves document as text file
+    - When true, converts document to text and saves to primary storage only (Azure or Local)
+    - Does not save to GCS
+    - Original document is deleted from storage after text conversion
+  - `hash` (optional): Unique identifier for the file
+  - `checkHash` (optional): Check if hash exists
+  - `clearHash` (optional): Remove hash from storage
+  - `fetch`/`load`/`restore` (optional): URL to fetch remote file (these are aliases - any of the three parameters will trigger the same remote file processing behavior)
+    - Does not require `requestId`
+    - Uses Redis caching
+    - Downloads and validates file first
+    - Ensures correct file extension
+    - Truncates long filenames
+- **Behavior**:
+  - For documents (PDF, DOC, etc.):
+    - If `save=true`: 
+      - Converts document to text
+      - Saves text file to primary storage (Azure or Local)
+      - Deletes original document from storage
+      - Does not save to GCS
+      - Returns object with primary storage URL
+    - If `save=false`: 
+      - Converts document to text
+      - Returns array of text chunks
+      - Does not persist any files
+  - For media files:
+    - Splits into chunks
+    - Uploads chunks to primary storage and GCS (if configured)
+    - Returns chunk information with offsets
+  - For remote files (`fetch`/`load`/`restore`):
+    - Downloads file from URL
+    - Processes based on file type
+    - Returns processed result
+    - Caches result in Redis using URL as key
+    - Updates Redis timestamp on subsequent requests
+    - Truncates filenames longer than 200 characters
+    - Ensures correct file extension based on content type
+
+### DELETE
+- **Purpose**: Remove files from storage
+- **Parameters** (can be in query string or request body):
+  - `requestId` (required): Unique identifier for the request
+- **Behavior**:
+  - Deletes file from primary storage (Azure or Local)
+  - Deletes file from GCS if configured
+  - Returns deletion result
+- **Response**: Array of deleted file URLs
+
+## Storage Configuration
+- **Azure**: Enabled if `AZURE_STORAGE_CONNECTION_STRING` is set
+- **GCS**: Enabled if `GCP_SERVICE_ACCOUNT_KEY_BASE64` or `GCP_SERVICE_ACCOUNT_KEY` is set
+- **Local**: Used as fallback if Azure is not configured
+
+## Response Format
+- **Success**: 
+  - Status: 200
+  - Body: Varies by operation (see specific methods above)
+- **Error**:
+  - Status: 400/404/500
+  - Body: Error message string
+
+## Progress Tracking
+- Progress updates are published to Redis for each operation
+- Progress includes:
+  - `progress`: Completion percentage (0-1)
+  - `completedCount`: Number of completed steps
+  - `totalCount`: Total number of steps
+  - `numberOfChunks`: Number of chunks (for media files)
+  - `data`: Additional operation data
+- Progress updates are published to Redis channel associated with `requestId`
+
+## File Types
+- **Documents**: Processed based on `DOC_EXTENSIONS` list
+  - Supported extensions: 
+    - Text: .txt, .json, .csv, .md, .xml, .js, .html, .css
+    - Office: .doc, .docx, .xls, .xlsx
+  - Document processing limitations:
+    - PDFs: Does not support scanned, encrypted, or password-protected PDFs
+    - Requires OCR for PDFs without embedded fonts
+  - Text chunking:
+    - Maximum chunk size: 10,000 characters
+    - Chunks are split at sentence boundaries when possible
+    - Returns array of text chunks
+- **Media**: All other file types, processed through chunking
+  - Chunked into smaller pieces for processing
+  - Each chunk is stored separately
+  - Media chunking behavior:
+    - Default chunk duration: 500 seconds
+    - Chunks are processed in parallel (3 at a time)
+    - Audio is converted to MP3 format (128kbps)
+    - Uses 4MB read buffer for file processing
+  - Supported media types:
+    - Images: .jpg, .jpeg, .png, .webp, .heic, .heif, .pdf
+    - Video: .mp4, .mpeg, .mov, .avi, .flv, .mpg, .webm, .wmv, .3gp
+    - Audio: .wav, .mp3, .aac, .ogg, .flac, .m4a
+  - File download behavior:
+    - 30 second timeout for downloads
+    - Supports streaming downloads
+    - Handles URL encoding/decoding
+    - Truncates filenames longer than 200 characters
+
+## Storage Behavior
+- **Primary Storage** (Azure or Local):
+  - Files are stored with UUID-based names
+  - Organized by requestId folders
+  - Azure: Uses SAS tokens for access
+  - Local: Served via HTTP on configured port
+- **GCS** (if configured):
+  - Files stored with gs:// protocol URLs
+  - Same folder structure as primary storage
+  - Only used for media file chunks
+- **Redis**:
+  - Stores file metadata and URLs
+  - Used for caching remote file results
+  - Tracks file access timestamps
+  - Used for progress tracking
+
+## Cleanup
+- Automatic cleanup of inactive files
+- Removes files from:
+  - Primary storage (Azure/Local)
+  - GCS (if configured)
+  - Redis file store map
+- Cleanup is triggered on each request but only runs if not already in progress
+- Temporary files are cleaned up:
+  - After 1 hour of inactivity
+  - After successful processing
+  - On error conditions
+
+## Error Handling
+- **400 Bad Request**: 
+  - Missing required parameters
+  - Invalid or inaccessible URL
+  - Unsupported file type
+- **404 Not Found**: 
+  - File or hash not found
+  - File not found in storage
+- **500 Internal Server Error**: 
+  - Processing errors
+  - Storage errors
+  - Document conversion errors
+  - PDF processing errors (scanned, encrypted, password-protected)
+- All errors include descriptive message in response body 

package/helper-apps/cortex-file-handler/function.json

@@ -5,11 +5,7 @@
       "type": "httpTrigger",
       "direction": "in",
       "name": "req",
-      "methods": [
-        "get",
-        "post",
-        "delete"
-      ]
+      "methods": ["get", "post", "delete"]
     },
     {
       "type": "http",
@@ -17,4 +13,4 @@
       "name": "res"
     }
   ]
-}
+}