@minded-ai/mindedjs 2.0.28 → 2.0.29-beta.1

package/docs/self-hosting/architecture.md

@@ -0,0 +1,403 @@
+# Self-Hosting Architecture
+
+This document explains how MindedJS self-hosting works, the architecture, and data flow between your infrastructure and Minded's platform.
+
+## Table of Contents
+
+- [Hybrid Architecture Overview](#hybrid-architecture-overview)
+- [Control Plane vs Agentic Plane](#control-plane-vs-agentic-plane)
+- [Data Flow & Security](#data-flow--security)
+- [Agent Development & Deployment](#agent-development--deployment)
+- [Data Sovereignty](#data-sovereignty)
+
+## Hybrid Architecture Overview
+
+MindedJS uses a **hybrid architecture** where agents run in your infrastructure while being developed and managed through the Minded platform.
+
+This model provides:
+
+- **Data Sovereignty**: All PII data stays in your infrastructure
+- **Developer Experience**: Use Minded's web IDE and platform features
+- **Zero Ops Burden**: Minded handles orchestration, monitoring, and management
+- **Enterprise Compliance**: Meet data residency requirements
+
+## Control Plane vs Agentic Plane
+
+### Control Plane (Minded-Hosted SaaS)
+
+The control plane is Minded's managed infrastructure that provides:
+
+- **Agent Studio**: Web IDE for building and configuring agents
+- **Git Integration**: Every change commits to agent repositories
+- **Orchestration**: Manages deployments across dev/staging/production environments
+- **Monitoring**: Centralized dashboards and session visibility
+- **Load Balancing**: Routes requests across your agent pods
+- **WebSocket Gateway**: Manages connections between users and agents
+
+**What's stored in control plane:**
+
+- Agent configurations and metadata
+- User accounts and permissions
+- Git repositories (code only, no data)
+- Deployment history and logs
+
+**What's NOT stored in control plane:**
+
+- PII data from sessions
+- Customer conversation history
+- Screenshots or files from agent executions
+
+### Agentic Plane (Customer Infrastructure)
+
+Deployed in your AWS account via Terraform:
+
+#### 1. **Amazon EKS**
+
+Kubernetes cluster running agent containers
+
+- Fargate profiles for compute
+- Multiple namespaces (dev, staging, production)
+- Auto-scaling based on load
+- Health checks and self-healing
+
+#### 2. **Amazon ECR**
+
+Private container registry for agent Docker images
+
+- One repository per agent
+- KMS encryption
+- Lifecycle policies for image cleanup
+
+#### 3. **Amazon DocumentDB**
+
+Session history and agent state storage
+
+- Serverless v2 (scales 0.5-2 ACU)
+- MongoDB 5.0 compatible
+- Accessed by Minded backend via AWS PrivateLink
+- Minded backend operates in **0-retention mode** (pass-through only)
+
+#### 4. **Amazon S3**
+
+Two buckets for different purposes:
+
+**Logs Bucket**: Infrastructure and audit logs
+
+- CloudWatch logs export
+- Terraform state (optional)
+- Audit trails
+
+**Agent Data Bucket**: Agent-generated data
+
+- Browser screenshots
+- Downloaded files
+- Agent execution logs
+- Direct access from agent pods via IRSA
+- Organized by agent: `/agent-id/session-id/...`
+
+#### 5. **AWS PrivateLink + NLB**
+
+Secure cross-account DocumentDB access
+
+- Network Load Balancer in front of DocumentDB
+- VPC Endpoint Service for private connectivity
+- No data traverses public internet
+
+#### 6. **KMS**
+
+Encryption keys for all data at rest
+
+- Separate keys for S3, DocumentDB, ECR
+- Customer-managed keys
+- Access controlled via IAM
+
+## Data Flow & Security
+
+### Agent Development
+
+```
+Developer → Minded Platform → Git Repository
+                ↓
+          Auto-commit changes
+```
+
+- Users develop agents in Minded platform (web UI)
+- Changes commit to Git automatically
+- All code versioned and auditable
+
+### Agent Deployment
+
+```
+Git → Minded Backend → Build Docker → Push to Customer ECR → Deploy to Customer EKS
+```
+
+When you click "Deploy" in Minded platform:
+
+1. **Build Phase** (Minded backend):
+
+   - Pulls agent code from Git
+   - Adds MindedJS SDK and dependencies
+   - Builds Docker image
+   - Pushes to your ECR using cross-account role
+
+2. **Deployment Phase** (Minded backend):
+
+   - Updates Kubernetes deployment in your EKS
+   - Uses blue-green deployment for zero downtime
+   - Creates/updates service, ingress, configmaps
+   - Scales pods based on configuration
+
+3. **Startup**:
+   - Pods start and connect via WebSocket to Minded backend
+   - Minded backend registers pods in load balancer pool
+
+### Runtime Operations
+
+```
+User Request → Minded Backend → Agent Pod (Customer EKS)
+                                    ↓
+                          Customer DocumentDB ← Minded Backend (0-retention)
+                                    ↓
+                          Customer S3 (Direct Write)
+```
+
+**Request Flow:**
+
+1. User sends message via Minded platform
+2. Minded backend routes to available agent pod
+3. Agent pod processes request
+
+**Data Storage:**
+
+- **Metadata** (non-PII): Sent via WebSocket, stored in Minded infrastructure
+  - Agent configuration
+  - Flow definitions
+  - Deployment status
+- **PII Data**: Stored ONLY in customer infrastructure
+  - **S3**: Direct write from pods (screenshots, files, logs)
+  - **DocumentDB**: Write via Minded backend in **0-retention mode**
+    - Minded backend acts as pass-through proxy
+    - No data persisted in Minded infrastructure
+    - All PII stored exclusively in customer DocumentDB
+
+### Session Querying
+
+```
+User Query → Minded Backend → Customer DocumentDB/S3 → Stream to Browser
+```
+
+- Users query past sessions in Minded platform
+- Minded backend retrieves data from customer's DocumentDB/S3
+- Data streams directly to user's browser
+- Never persisted in Minded infrastructure
+
+### Security Guarantees
+
+1. **Data Isolation**: PII never leaves customer AWS account
+2. **Encryption**: All data encrypted at rest (KMS) and in transit (TLS)
+3. **Zero Retention**: Minded backend doesn't persist PII data
+4. **Private Connectivity**: DocumentDB accessed via PrivateLink (no public internet)
+5. **IAM Controls**: Least-privilege access via cross-account roles
+6. **Audit Trails**: All access logged in CloudWatch
+
+## Agent Development & Deployment
+
+### Development Workflow
+
+```
+1. Create Agent (Minded UI)
+   ↓
+2. Configure (Flows, Tools, Prompts)
+   ↓
+3. Auto-Deploy to Dev
+   ↓
+4. Test in Dev Environment
+   ↓
+5. Promote to Staging
+   ↓
+6. Promote to Production
+```
+
+**Step-by-step:**
+
+1. **Create Agent**: In Minded platform UI
+
+   - Give it a name and initial configuration
+   - Agent repository created in Git
+
+2. **Configure Agent**: Set up flows, tools, prompts
+
+   - Visual flow builder
+   - Configure integrations
+   - Add custom tools
+
+3. **Auto-Deploy to Dev**: Changes automatically deploy to `dev` namespace
+
+   - Every save triggers Git commit
+   - Automatic build and deploy to dev
+   - Instant feedback loop
+
+4. **Test**: Run test sessions in dev environment
+
+   - Interactive testing in platform
+   - View logs and session data
+   - Debug and iterate
+
+5. **Promote to Staging**: Manual promotion via UI
+
+   - Click "Publish to Staging"
+   - Immutable deployment from specific Git commit
+   - Test in staging environment
+
+6. **Promote to Production**: Manual promotion after staging validation
+   - Click "Publish to Production"
+   - Blue-green deployment for zero downtime
+   - Rollback capability
+
+### Deployment Details
+
+**Build Phase** (Minded Backend):
+
+- Pulls agent code from Git repository
+- Injects MindedJS SDK and runtime
+- Adds dependencies from package.json
+- Builds Docker image
+- Tags with environment and version
+- Pushes to customer's ECR via cross-account role
+
+**Deployment Phase** (Minded Backend):
+
+- Assumes cross-account role in customer AWS
+- Updates Kubernetes deployment manifest
+- Applies to EKS cluster in target namespace
+- Waits for pods to become healthy
+- Updates service routing
+
+**Blue-Green Deployment:**
+
+1. New pods start alongside old pods
+2. Health checks verify new pods ready
+3. Traffic switches to new pods
+4. Old pods gracefully shut down
+5. Zero downtime for end users
+
+### Multiple Agents
+
+You can run multiple agents in the same infrastructure. Each agent gets:
+
+- **Dedicated ECR repository**: Independent versioning and deployment
+- **Kubernetes service with 3 namespaces**: dev, staging, production
+- **Independent scaling**: Each agent scales based on its own load
+- **Isolated data storage**: Separate paths in shared S3 bucket
+  - Agent A: `/agent-123/...`
+  - Agent B: `/agent-456/...`
+
+**Shared Resources:**
+
+- EKS cluster (all agents run as pods)
+- DocumentDB cluster (separate databases or collections)
+- S3 buckets (separate prefixes)
+- VPC and networking
+
+## Data Sovereignty
+
+### What Data Lives Where
+
+| Data Type           | Location            | Retention           | Access               |
+| ------------------- | ------------------- | ------------------- | -------------------- |
+| Agent code & config | Minded Git          | Indefinite          | Minded platform      |
+| User accounts       | Minded DB           | Indefinite          | Minded platform      |
+| Session metadata    | Minded DB           | Configurable        | Minded platform      |
+| Conversation PII    | Customer DocumentDB | Customer controlled | Via PrivateLink      |
+| Screenshots & files | Customer S3         | Customer controlled | IRSA + Cross-account |
+| Agent logs          | Customer S3         | Customer controlled | IRSA + Cross-account |
+| Infrastructure logs | Customer CloudWatch | Customer controlled | Customer only        |
+
+### Compliance Benefits
+
+**GDPR Compliance:**
+
+- All personal data stored in customer-controlled infrastructure
+- Customer owns data deletion and retention policies
+- Right to be forgotten: Delete data directly from your DocumentDB/S3
+
+**Data Residency:**
+
+- Deploy in any AWS region
+- Data never leaves your chosen region
+- Meets local data residency requirements
+
+**SOC 2 / ISO 27001:**
+
+- Audit trails in your CloudWatch
+- Encryption at rest and in transit
+- Least-privilege IAM policies
+- Private network connectivity
+
+### Zero-Retention Model
+
+Minded backend operates in **0-retention mode** for PII:
+
+1. **Write Operations:**
+
+   ```
+   Agent Pod → WebSocket → Minded Backend → PrivateLink → Customer DocumentDB
+   ```
+
+   - Minded backend receives data via WebSocket
+   - Immediately writes to customer DocumentDB
+   - Does NOT persist in Minded infrastructure
+   - Returns confirmation to agent pod
+
+2. **Read Operations:**
+   ```
+   User Query → Minded Backend → PrivateLink → Customer DocumentDB → Stream to Browser
+   ```
+   - Minded backend retrieves from customer DocumentDB
+   - Streams directly to user's browser
+   - Does NOT cache or persist
+   - Connection closed after streaming
+
+This ensures Minded never stores customer PII data, only metadata needed for orchestration.
+
+## Infrastructure Ownership
+
+### Customer Owns & Controls
+
+- **All infrastructure resources**: Full AWS account access
+- **All data**: DocumentDB, S3, logs
+- **Encryption keys**: Customer-managed KMS keys
+- **Retention policies**: Configure backups and lifecycle
+- **Cost**: Direct AWS billing to customer
+- **Scaling**: Adjust cluster size and limits
+- **Security**: VPC configuration, security groups
+
+### Minded Manages
+
+- **Agent deployments**: Automated via cross-account role
+- **Platform features**: Studio, monitoring, orchestration
+- **SDK updates**: New features and improvements
+- **Load balancing**: Routing across agent pods
+- **Health monitoring**: Pod health and auto-restart
+
+### Shared Responsibility
+
+| Aspect                      | Minded        | Customer          |
+| --------------------------- | ------------- | ----------------- |
+| Agent code development      | ✓ Platform UI | ✗                 |
+| Agent deployment            | ✓ Automation  | ✗                 |
+| Infrastructure provisioning | ✗             | ✓ Terraform       |
+| Infrastructure maintenance  | ✗             | ✓ Updates/patches |
+| Data storage                | ✗             | ✓ S3/DocumentDB   |
+| Data retention              | ✗             | ✓ Policies        |
+| Encryption keys             | ✗             | ✓ KMS             |
+| Cost                        | ✗             | ✓ AWS billing     |
+| Monitoring infrastructure   | ✗             | ✓ CloudWatch      |
+| Monitoring agent sessions   | ✓ Platform    | ✓ CloudWatch      |
+
+## Next Steps
+
+- [AWS Deployment Guide](./aws.md) - Step-by-step Terraform deployment
+- [Examples](./examples/README.md) - Reference configurations
+- [README](./README.md) - General self-hosting overview
+