traia-iatp 0.1.29__py3-none-any.whl

traia_iatp/registry/readmes/ATLAS_SEARCH_INDEXES.md

@@ -0,0 +1,252 @@
+# MongoDB Atlas Search Indexes for IATP Registry
+
+This document describes the MongoDB Atlas Search and Vector Search indexes required for the IATP utility agent registry.
+
+## Prerequisites
+
+1. MongoDB Atlas cluster (M10 or higher for Vector Search)
+2. Access to Atlas UI or Atlas Admin API
+3. OpenAI API key for generating embeddings (set as `OPENAI_API_KEY` environment variable)
+
+## Index Configuration
+
+### 1. Utility Agent Atlas Search Index
+
+Create this index on the `utility_agents` collection (or `utility_agents_test`/`utility_agents_prod` based on environment).
+
+**Index Name**: `utility_agent_atlas_search` (or `utility_agent_atlas_search_test`/`utility_agent_atlas_search_prod`)
+
+**Index Definition**:
+```json
+{
+  "mappings": {
+    "dynamic": false,
+    "fields": {
+      "name": {
+        "type": "string",
+        "analyzer": "lucene.standard"
+      },
+      "description": {
+        "type": "string",
+        "analyzer": "lucene.standard"
+      },
+      "tags": {
+        "type": "string",
+        "analyzer": "lucene.standard"
+      },
+      "capabilities": {
+        "type": "string",
+        "analyzer": "lucene.standard"
+      },
+      "search_text": {
+        "type": "string",
+        "analyzer": "lucene.standard"
+      },
+      "skills": {
+        "type": "document",
+        "fields": {
+          "name": {
+            "type": "string",
+            "analyzer": "lucene.standard"
+          },
+          "description": {
+            "type": "string",
+            "analyzer": "lucene.standard"
+          },
+          "examples": {
+            "type": "string",
+            "analyzer": "lucene.standard"
+          }
+        }
+      },
+      "agent_card": {
+        "type": "document",
+        "fields": {
+          "name": {
+            "type": "string",
+            "analyzer": "lucene.standard"
+          },
+          "description": {
+            "type": "string",
+            "analyzer": "lucene.standard"
+          }
+        }
+      }
+    }
+  }
+}
+```
+
+### 2. Utility Agent Vector Search Index
+
+Create this index for semantic search using embeddings.
+
+**Index Name**: `utility_agent_vector_search` (or `utility_agent_vector_search_test`/`utility_agent_vector_search_prod`)
+
+**Index Definition**:
+```json
+{
+  "type": "vectorSearch",
+  "fields": [
+    {
+      "type": "vector",
+      "path": "embeddings.search_text",
+      "numDimensions": 1536,
+      "similarity": "cosine"
+    },
+    {
+      "type": "vector",
+      "path": "embeddings.description",
+      "numDimensions": 1536,
+      "similarity": "cosine"
+    },
+    {
+      "type": "vector",
+      "path": "embeddings.capabilities",
+      "numDimensions": 1536,
+      "similarity": "cosine"
+    },
+    {
+      "type": "vector",
+      "path": "embeddings.skills",
+      "numDimensions": 1536,
+      "similarity": "cosine"
+    },
+    {
+      "type": "filter",
+      "path": "is_active"
+    },
+    {
+      "type": "filter",
+      "path": "tags"
+    }
+  ]
+}
+```
+
+### 3. MCP Server Atlas Search Index
+
+Create this index on the `mcp_servers` collection.
+
+**Index Name**: `mcp_server_atlas_search` (or `mcp_server_atlas_search_test`/`mcp_server_atlas_search_prod`)
+
+**Index Definition**:
+```json
+{
+  "mappings": {
+    "dynamic": false,
+    "fields": {
+      "name": {
+        "type": "string",
+        "analyzer": "lucene.standard"
+      },
+      "description": {
+        "type": "string",
+        "analyzer": "lucene.standard"
+      },
+      "capabilities": {
+        "type": "string",
+        "analyzer": "lucene.standard"
+      }
+    }
+  }
+}
+```
+
+### 4. MCP Server Vector Search Index
+
+**Index Name**: `mcp_server_vector_search` (or `mcp_server_vector_search_test`/`mcp_server_vector_search_prod`)
+
+**Index Definition**:
+```json
+{
+  "type": "vectorSearch",
+  "fields": [
+    {
+      "type": "vector",
+      "path": "description_embedding",
+      "numDimensions": 1536,
+      "similarity": "cosine"
+    },
+    {
+      "type": "vector",
+      "path": "capabilities_embedding",
+      "numDimensions": 1536,
+      "similarity": "cosine"
+    },
+    {
+      "type": "filter",
+      "path": "is_active"
+    }
+  ]
+}
+```
+
+## Creating Indexes
+
+### Via Atlas UI
+
+1. Navigate to your cluster in MongoDB Atlas
+2. Go to "Search" tab
+3. Click "Create Search Index"
+4. Choose "JSON Editor"
+5. Select the appropriate database and collection
+6. Paste the index definition
+7. Name the index according to the pattern above
+8. Click "Create Search Index"
+
+### Via Atlas Admin API
+
+```bash
+# Set your API credentials
+export ATLAS_PUBLIC_KEY="your-public-key"
+export ATLAS_PRIVATE_KEY="your-private-key"
+export ATLAS_PROJECT_ID="your-project-id"
+export ATLAS_CLUSTER_NAME="your-cluster-name"
+
+# Create utility agent Atlas Search index
+curl --user "${ATLAS_PUBLIC_KEY}:${ATLAS_PRIVATE_KEY}" --digest \
+  --header "Content-Type: application/json" \
+  --request POST "https://cloud.mongodb.com/api/atlas/v2/groups/${ATLAS_PROJECT_ID}/clusters/${ATLAS_CLUSTER_NAME}/search/indexes" \
+  --data '{
+    "collectionName": "utility_agents",
+    "database": "iatp",
+    "name": "utility_agent_atlas_search",
+    "mappings": { ... }
+  }'
+```
+
+## Index Naming Convention
+
+The indexes follow environment-specific naming:
+- Test environment: `*_test` suffix
+- Production environment: `*_prod` suffix
+- Default/development: No suffix
+
+## Required Embedding Dimensions
+
+All vector fields use OpenAI's `text-embedding-3-small` model which produces 1536-dimensional vectors.
+
+## Monitoring Index Creation
+
+Index creation can take several minutes. Monitor progress:
+1. In Atlas UI: Check the "Search" tab for index status
+2. Via API: Query the index endpoint to check status
+
+## Testing Indexes
+
+After creation, test the indexes using the registry methods:
+
+```python
+# Test Atlas Search
+results = await registry.atlas_search("trading hyperliquid")
+
+# Test Vector Search
+results = await registry.vector_search_text("find me a trading agent for crypto")
+```
+
+## Maintenance
+
+- Indexes are automatically maintained by Atlas
+- No manual optimization required
+- Monitor index performance in Atlas UI under "Search Metrics" 

traia_iatp/registry/readmes/ATLAS_SEARCH_SETUP.md

@@ -0,0 +1,134 @@
+# MongoDB Atlas Search Index Setup
+
+This guide explains how to set up Atlas Search and Vector Search indexes for the IATP registry collections.
+
+## Overview
+
+Each collection requires two types of indexes:
+1. **Atlas Search Index** - For text-based keyword search
+2. **Vector Search Index** - For semantic similarity search using embeddings
+
+## Index Naming Convention
+
+All index names include the environment suffix:
+- `utility_agency_atlas_search_{env}`
+- `utility_agency_vector_search_{env}`
+- `mcp_server_atlas_search_{env}`
+- `mcp_server_vector_search_{env}`
+
+Where `{env}` is one of: `test`, `staging`, or `prod`
+
+## Collections
+
+Based on your environment (ENV variable), the collections are:
+- `iatp-utility-agency-registry-{env}` (where env = test/staging/prod)
+- `iatp-mcp-server-registry-{env}`
+
+## Creating Indexes in MongoDB Atlas
+
+### Method 1: Atlas UI
+
+1. Go to your MongoDB Atlas cluster
+2. Click on "Search" in the left sidebar
+3. Click "Create Search Index"
+4. Choose your database (`iatp`)
+5. For each collection, create both indexes:
+
+#### Utility Agency Registry - Atlas Search Index
+- Collection: `iatp-utility-agency-registry-{env}`
+- Index Name: `utility_agency_atlas_search_{env}`
+- Use the JSON definition from `atlas_search_indexes.json` under `utility_agency_indexes.atlas_search`
+
+#### Utility Agency Registry - Vector Search Index
+- Collection: `iatp-utility-agency-registry-{env}`
+- Index Name: `utility_agency_vector_search_{env}`
+- Use the JSON definition from `atlas_search_indexes.json` under `utility_agency_indexes.vector_search`
+
+#### MCP Server Registry - Atlas Search Index
+- Collection: `iatp-mcp-server-registry-{env}`
+- Index Name: `mcp_server_atlas_search_{env}`
+- Use the JSON definition from `atlas_search_indexes.json` under `mcp_server_indexes.atlas_search`
+
+#### MCP Server Registry - Vector Search Index
+- Collection: `iatp-mcp-server-registry-{env}`
+- Index Name: `mcp_server_vector_search_{env}`
+- Use the JSON definition from `atlas_search_indexes.json` under `mcp_server_indexes.vector_search`
+
+### Method 2: Atlas Admin API
+
+```bash
+# Set your Atlas API credentials
+export ATLAS_PUBLIC_KEY="your_public_key"
+export ATLAS_PRIVATE_KEY="your_private_key"
+export ATLAS_PROJECT_ID="your_project_id"
+export ATLAS_CLUSTER_NAME="your_cluster_name"
+export ENV="test"  # or staging, prod
+
+# Create indexes using curl (example for one index)
+# Note: You'll need to modify the JSON to replace <env> with your actual environment
+curl --user "${ATLAS_PUBLIC_KEY}:${ATLAS_PRIVATE_KEY}" --digest \
+  --header "Content-Type: application/json" \
+  --request POST "https://cloud.mongodb.com/api/atlas/v1.0/groups/${ATLAS_PROJECT_ID}/clusters/${ATLAS_CLUSTER_NAME}/fts/indexes" \
+  --data @atlas_search_indexes.json
+```
+
+## Example: Creating Indexes for Test Environment
+
+For the `test` environment, you would create:
+- `utility_agency_atlas_search_test`
+- `utility_agency_vector_search_test`
+- `mcp_server_atlas_search_test`
+- `mcp_server_vector_search_test`
+
+## Vector Embeddings
+
+The vector search indexes expect the following embedding fields:
+- `description_embedding` - 1536-dimensional vector for description text
+- `tags_embedding` - 1536-dimensional vector for tags (utility agencies)
+- `capabilities_embedding` - 1536-dimensional vector for capabilities (MCP servers)
+
+### Generating Embeddings
+
+To use vector search, you need to generate embeddings when adding documents. Example using OpenAI:
+
+```python
+import openai
+
+def generate_embedding(text: str) -> List[float]:
+    response = openai.Embedding.create(
+        model="text-embedding-ada-002",
+        input=text
+    )
+    return response['data'][0]['embedding']
+```
+
+## Using the Indexes
+
+The code automatically uses the correct index name based on your ENV variable:
+
+### Atlas Search (Text Search)
+```python
+# The atlas_search method automatically uses the correct index name
+results = await registry.atlas_search("weather forecast")
+```
+
+### Vector Search (Semantic Search)
+```python
+# The vector_search method automatically uses the correct index name
+query_embedding = generate_embedding("meteorological predictions")
+results = await registry.vector_search(query_embedding)
+```
+
+## Important Notes
+
+1. **Index Status**: After creating indexes, they may take a few minutes to build
+2. **Environment Variable**: Make sure your `ENV` environment variable is set correctly
+3. **Embedding Dimensions**: The vector indexes are configured for 1536 dimensions (OpenAI's ada-002 model)
+4. **Similarity Metric**: Using cosine similarity for vector comparison
+5. **Filters**: The vector indexes include `is_active` as a filter field for efficient filtering
+
+## Monitoring
+
+You can monitor index status and usage in the Atlas UI under:
+- Search > Index Status
+- Search > Query Analytics 

traia_iatp/registry/readmes/AUTHENTICATION_UPDATE.md

@@ -0,0 +1,124 @@
+# MongoDB Authentication Update Summary
+
+## Overview
+
+We have simplified MongoDB authentication by removing API key support and focusing on two primary authentication methods:
+
+1. **X.509 Certificate Authentication** (Recommended)
+2. **Username/Password Authentication** (Fallback)
+
+## Changes Made
+
+### 1. Code Updates
+
+#### mongodb_registry.py
+- Removed all references to `MONGODB_READWRITE_API_KEY` and `MONGODB_READONLY_API_KEY`
+- Updated both `UtilityAgentRegistry` and `MCPServerRegistry` classes
+- Simplified authentication priority to: Certificate → Username/Password → Connection String
+
+#### iatp_registry_api.py
+- Removed API key authentication from `get_readonly_connection_string()`
+- Updated error messages to reflect new authentication methods
+
+### 2. Removed Files
+- `mongodb_data_api_client.py` - MongoDB Data API client (API key based)
+- `update_env_for_api_keys.py` - API key setup utility
+- `test_end_to_end_with_api_keys.py` - API key specific test
+
+### 3. Documentation Updates
+- Updated `MONGODB_X509_AUTH.md` to reflect simplified authentication priority
+- Updated `README.md` files to remove API key references
+- Updated `search_api_service.py` documentation
+
+### 4. Test Updates
+- Updated `test_mongodb_x509_auth.py` to remove API key checks
+- Created `test_end_to_end_with_auth.py` as replacement for API key test
+
+## Authentication Methods
+
+### Primary: X.509 Certificate
+```bash
+export MONGODB_X509_CERT_FILE=/path/to/certificate.pem
+```
+
+**Benefits:**
+- Most secure - no passwords in environment
+- Certificate-based mutual TLS authentication
+- Can be rotated without code changes
+- Supported by MongoDB Atlas
+
+### Fallback: Username/Password
+```bash
+export MONGODB_USER=username
+export MONGODB_PASSWORD=password
+```
+
+**When to use:**
+- Development environments
+- Legacy systems
+- When certificates are not available
+
+### Connection String
+```bash
+export MONGODB_CONNECTION_STRING=mongodb+srv://...
+```
+
+**When to use:**
+- Complex connection configurations
+- Custom authentication mechanisms
+
+## Migration Guide
+
+### From API Keys to X.509 Certificates
+
+1. Create X.509 user in MongoDB Atlas
+2. Download certificate file (.pem)
+3. Update environment:
+   ```bash
+   # Remove
+   unset MONGODB_READWRITE_API_KEY
+   unset MONGODB_READONLY_API_KEY
+   
+   # Add
+   export MONGODB_X509_CERT_FILE=/path/to/cert.pem
+   ```
+4. Test with: `uv run python tests/test_mongodb_x509_auth.py`
+
+### From API Keys to Username/Password
+
+1. Update environment:
+   ```bash
+   # Remove
+   unset MONGODB_READWRITE_API_KEY
+   unset MONGODB_READONLY_API_KEY
+   
+   # Add
+   export MONGODB_USER=your-username
+   export MONGODB_PASSWORD=your-password
+   ```
+
+## Testing
+
+Run authentication tests:
+```bash
+# Test X.509 authentication
+uv run python tests/test_mongodb_registry/test_mongodb_x509_auth.py
+
+# Test end-to-end with current auth method
+uv run python tests/test_mongodb_registry/test_end_to_end_with_auth.py
+```
+
+## Security Recommendations
+
+1. **Use X.509 certificates in production** - Most secure option
+2. **Protect certificate files** - Set file permissions to 600
+3. **Rotate credentials regularly** - Both certificates and passwords
+4. **Use environment-specific credentials** - Different for dev/staging/prod
+5. **Never commit credentials** - Use secure secret management
+
+## Backward Compatibility
+
+The system maintains backward compatibility:
+- Existing username/password authentication continues to work
+- Connection strings are still supported
+- The authentication priority ensures smooth transition 

traia_iatp/registry/readmes/EMBEDDINGS_SETUP.md

@@ -0,0 +1,172 @@
+# Embeddings Setup for Vector Search
+
+This guide explains how to enable automatic embedding generation for vector search in IATP.
+
+## Important Note
+
+MongoDB Atlas does **not** automatically generate embeddings. The IATP application handles embedding generation:
+- When documents are added, IATP generates embeddings using your chosen provider (OpenAI, Cohere, etc.)
+- When searching, IATP converts your text query to embeddings before sending to MongoDB
+- This happens transparently when `ENABLE_EMBEDDINGS=true`
+
+## Overview
+
+IATP can automatically generate embeddings when adding documents to the registry, enabling semantic vector search. This is optional - the system works without embeddings, but you won't be able to use vector search.
+
+## Environment Variables
+
+To enable embeddings, set the following environment variables:
+
+```bash
+# Enable embedding generation
+export ENABLE_EMBEDDINGS=true
+
+# Choose your embedding provider (default: openai)
+# Options: openai, cohere
+export EMBEDDING_PROVIDER=openai
+
+# Provider-specific API keys
+export OPENAI_API_KEY=your_openai_api_key      # For OpenAI
+export COHERE_API_KEY=your_cohere_api_key      # For Cohere
+```
+
+## How It Works
+
+### 1. Document Storage with Embeddings
+
+When `ENABLE_EMBEDDINGS=true`, the system automatically generates embeddings:
+
+**For Utility Agencies:**
+- `description_embedding`: Vector embedding of the description text
+- `tags_embedding`: Vector embedding of concatenated tags
+
+**For MCP Servers:**
+- `description_embedding`: Vector embedding of the description text
+- `capabilities_embedding`: Vector embedding of concatenated capabilities
+
+### 2. Search Methods
+
+**Without Embeddings (Always Available):**
+```python
+# Basic keyword search (requires MongoDB text index)
+results = await registry.query_agencies(query="weather")
+
+# Atlas Search (requires Atlas Search index)
+results = await registry.atlas_search("weather forecast")
+```
+
+**With Embeddings (Requires ENABLE_EMBEDDINGS=true):**
+```python
+# Vector search with text query (auto-converts to embedding)
+results = await registry.vector_search_text("meteorological predictions")
+
+# Vector search with specific field
+results = await registry.vector_search_text("api integration", search_field="tags")
+
+# Vector search with pre-computed embedding
+embedding = await embedding_service.generate_embedding("weather")
+results = await registry.vector_search(embedding)
+```
+
+## Embedding Providers
+
+### OpenAI (Default)
+- Model: `text-embedding-ada-002` (1536 dimensions)
+- Best for: General-purpose semantic search
+- Cost: ~$0.0001 per 1K tokens
+
+### Cohere
+- Model: `embed-english-v3.0` (1024 dimensions)
+- Best for: Multilingual support
+- Note: Update vector index dimensions to 1024 if using Cohere
+
+## Installation
+
+IATP includes optional dependencies for embeddings support. Install using uv:
+
+```bash
+# Install IATP with embeddings support (includes all providers)
+uv sync --extra embeddings
+
+# Or add specific providers to your project
+uv add openai      # For OpenAI embeddings
+uv add cohere      # For Cohere embeddings
+```
+
+The embedding providers are defined as optional dependencies in `pyproject.toml`:
+```toml
+[project.optional-dependencies]
+embeddings = [
+    "openai>=1.0.0",
+    "cohere>=5.0.0",
+]
+```
+
+### Development Note
+
+When developing with uv:
+- Use `uv sync --extra embeddings` to install optional dependencies
+- Use `uv add <package>` to add new dependencies to your project
+- Dependencies added with `uv add` will be added to the main dependencies in `pyproject.toml`
+
+## Example Usage
+
+```python
+import os
+os.environ["ENABLE_EMBEDDINGS"] = "true"
+os.environ["OPENAI_API_KEY"] = "your-key"
+
+from traia_iatp.registry.mongodb_registry import UtilityAgencyRegistry
+
+# Create registry (embeddings will be auto-generated)
+registry = UtilityAgencyRegistry()
+
+# Add agency - embeddings generated automatically
+await registry.add_agency(agency, endpoint, tags=["weather", "api"])
+
+# Search using semantic similarity
+results = await registry.vector_search_text("climate data analysis")
+```
+
+## Performance Considerations
+
+1. **Embedding Generation Time**: ~100-500ms per text
+2. **Batch Processing**: Use `generate_embeddings()` for multiple texts
+3. **Caching**: Consider caching embeddings for frequently searched queries
+4. **Cost**: Monitor API usage, especially for large datasets
+
+## Troubleshooting
+
+### "Failed to generate embeddings" Warning
+- Check that `ENABLE_EMBEDDINGS=true`
+- Verify API key is set correctly
+- Check network connectivity
+- Verify provider package is installed
+
+### Vector Search Returns No Results
+- Ensure documents have embeddings (added after enabling embeddings)
+- Verify Vector Search index exists in Atlas
+- Check that embedding dimensions match (1536 for OpenAI)
+
+### Re-indexing Existing Data
+If you have existing data without embeddings:
+
+```python
+# Script to add embeddings to existing documents
+async def reindex_with_embeddings():
+    registry = UtilityAgencyRegistry()
+    
+    # Get all agencies
+    all_agencies = await registry.query_agencies(limit=1000)
+    
+    for agency in all_agencies:
+        # Re-add to generate embeddings
+        await registry.add_agency(agency, agency.endpoint, agency.tags)
+```
+
+## Security Notes
+
+1. **API Keys**: Never commit API keys to version control
+2. **Rate Limits**: Implement rate limiting for production
+3. **Error Handling**: System continues without embeddings if generation fails
+4. **Data Privacy**: Be aware that text is sent to external APIs for embedding