@minded-ai/mindedjs 2.0.28 → 2.0.29-beta.1

package/docs/self-hosting/aws.md

@@ -0,0 +1,338 @@
+# AWS Deployment
+
+Step-by-step guide to deploy MindedJS agents on AWS EKS using Terraform.
+
+## Table of Contents
+
+- [Prerequisites](#prerequisites)
+- [Setup Process](#setup-process)
+- [Terraform Configuration](#terraform-configuration)
+- [Deployment](#deployment)
+- [Verification](#verification)
+- [Outputs](#outputs)
+- [Next Steps](#next-steps)
+
+## Prerequisites
+
+### Required Tools
+
+- **Terraform** (v1.0+)
+- **AWS CLI** (v2+)
+- **kubectl** (for verification)
+- **Git**
+
+### AWS Requirements
+
+- AWS account with appropriate permissions
+- Ability to create:
+  - VPC and networking resources
+  - EKS clusters
+  - DocumentDB clusters
+  - S3 buckets
+  - IAM roles and policies
+  - KMS keys
+
+### Minded Requirements
+
+1. Sign up at [minded.ai](https://minded.ai)
+2. Contact Minded team (support@minded.ai) with your AWS Account ID and region
+3. Receive from Minded:
+   - Minded AWS Account ID
+   - Minded AWS Region
+   - External ID (for cross-account role security)
+
+## Setup Process
+
+Here’s your section rewritten to use a **read-only deploy key** instead of direct HTTPS cloning:
+
+---
+
+### 1. Clone Terraform Repository
+
+1. **Receive a deploy key** (SSH public key) from the Minded team.
+   Save the private key locally (e.g. `~/.ssh/minded_terraform`).
+
+2. **Add the key to your SSH agent:**
+
+   ```bash
+   eval "$(ssh-agent -s)"
+   ssh-add ~/.ssh/minded_terraform
+   ```
+
+3. **Clone the repository using SSH:**
+
+   ```bash
+   git clone git@github.com:minded-ai/self-hosting-terraform.git
+   cd self-hosting-terraform/terraform
+   ```
+
+### 2. Configure Terraform Variables
+
+Create `terraform.tfvars` from the example:
+
+```bash
+cp terraform.tfvars.example terraform.tfvars
+```
+
+Edit `terraform.tfvars` with your values:
+
+```hcl
+# Your AWS Configuration
+aws_account = "123456789012"              # Your AWS Account ID
+region      = "us-east-1"                 # Your preferred AWS region
+customer    = "your-company-name"         # Your company identifier
+environment = "production"                # Environment name
+
+# Network Configuration
+cidr             = "10.0.0.0/16"          # VPC CIDR block
+max_subnet_count = 2                      # Subnets per AZ (recommend 2 for HA)
+
+# Retention Policies
+log_retention_in_days     = 90            # CloudWatch logs retention
+backups_retention_in_days = 30            # DocumentDB backups retention
+
+# Maintenance Windows (UTC)
+aws_maintenance_window = "sat:00:00-sat:02:00"
+aws_backup_window      = "03:00-07:00"
+
+# Access Control
+kms_administrators = [
+  "arn:aws:iam::123456789012:user/admin",
+  "arn:aws:iam::123456789012:role/admin-role"
+]
+
+eks_administrators = [
+  "arn:aws:iam::123456789012:user/admin",
+  "arn:aws:iam::123456789012:role/admin-role"
+]
+
+# Minded Configuration (provided by Minded team)
+minded_aws_account = "999999999999"       # Minded AWS Account ID
+minded_region      = "us-east-1"          # Minded AWS Region
+external_id        = "unique-external-id" # Security token from Minded
+
+# Agent Environments
+minded_environments = ["dev", "stg", "prod"]  # Kubernetes namespaces
+```
+
+### 3. Configure Terraform Backend (Recommended)
+
+Terraform automatically creates S3 bucket and DynamoDB table for remote state storage. After your first `terraform apply`, you'll receive the backend configuration in the outputs:
+
+```hcl
+terraform_backend = {
+  "bucket"         = "minded-production-dev-test-remote-state"
+  "dynamodb_table" = "minded-production-dev-test-remote-state"
+  "key"            = "terraform/production/dev-test/terraform.tfstate"
+  "kms_key_id"     = "c145dc0d-476c-46ca-82a0-836dfeb9f4b2"
+  "region"         = "us-east-1"
+}
+```
+
+To migrate to remote state:
+
+1. Copy the output values to `backend.tf`:
+
+```hcl
+terraform {
+  backend "s3" {
+    region         = "us-east-1"
+    bucket         = "minded-production-dev-test-remote-state"
+    key            = "terraform/production/dev-test/terraform.tfstate"
+    dynamodb_table = "minded-production-dev-test-remote-state"
+    encrypt        = true
+    kms_key_id     = "c145dc0d-476c-46ca-82a0-836dfeb9f4b2"
+  }
+}
+```
+
+2. Reinitialize Terraform:
+
+```bash
+terraform init
+```
+
+3. When prompted, type `yes` to copy the state from local file to S3 backend
+
+## Terraform Configuration
+
+### Key Components Deployed
+
+#### VPC & Networking
+
+- VPC with public and private subnets across multiple AZs
+- NAT Gateways for outbound internet access
+- S3 VPC Endpoint (gateway) for efficient S3 access
+- Security groups for EKS, DocumentDB, and NLB
+
+#### EKS Cluster
+
+- Kubernetes version 1.34
+- Fargate profiles for:
+  - `kube-system` namespace (core Kubernetes services)
+  - Customer namespaces (dev, stg, prod)
+- OIDC provider for IRSA
+- Managed add-ons: vpc-cni, kube-proxy, coredns
+- CloudWatch logging enabled
+
+#### DocumentDB
+
+- Serverless v2 configuration (scales 0.5-2 ACU)
+- Engine: MongoDB 5.0 compatible
+- Encryption at rest with KMS
+- Automated backups
+- Audit logging to CloudWatch
+- Network Load Balancer for PrivateLink access
+- VPC Endpoint Service for Minded backend access
+
+#### S3 Buckets
+
+1. **Logs Bucket**: Infrastructure logs, audit trails
+2. **Agent Data Bucket**: Agent-generated data, screenshots, files
+   - **Single shared bucket** for all agents
+   - Data organized by agent using prefixes: `/agent-id/session-id/...`
+
+Both buckets configured with:
+
+- KMS encryption
+- Versioning enabled
+- Public access blocked
+- SSL-only access
+
+#### ECR Repository
+
+- Private registry for agent Docker images
+- KMS encryption
+- Mutable tags (for iterative development)
+
+#### IAM Roles
+
+**Cross-Account Backend Role**: Allows Minded backend to:
+
+- Push Docker images to ECR
+- Deploy and manage EKS services
+- Read/write S3 agent data
+- Access DocumentDB via PrivateLink
+
+**Kubernetes Service Account Role** (IRSA): Allows agent pods to:
+
+- Write to S3 agent data bucket
+- Access KMS for encryption/decryption
+
+## Deployment
+
+### Initialize Terraform
+
+```bash
+terraform init
+```
+
+### Review Planned Changes
+
+```bash
+terraform plan
+```
+
+Review the output carefully. Terraform will create ~50-60 resources.
+
+### Apply Configuration
+
+```bash
+terraform apply
+```
+
+Type `yes` when prompted. Deployment takes approximately 20-30 minutes.
+
+### Monitor Progress
+
+```bash
+# Watch EKS cluster creation
+aws eks describe-cluster --name <cluster-name> --query 'cluster.status'
+
+# Watch DocumentDB cluster creation
+aws docdb describe-db-clusters --query 'DBClusters[0].Status'
+```
+
+## Verification
+
+### 1. Verify Infrastructure
+
+```bash
+# Get Terraform outputs
+terraform output
+
+# Verify EKS cluster
+aws eks describe-cluster --name <cluster-name>
+
+# Configure kubectl
+aws eks update-kubeconfig --name <cluster-name> --region <region>
+
+# Check nodes (Fargate)
+kubectl get nodes
+
+# Check namespaces
+kubectl get namespaces
+```
+
+### 2. Verify DocumentDB
+
+```bash
+# Check cluster status
+aws docdb describe-db-clusters
+
+# Get VPC Endpoint Service name (share with Minded)
+terraform output agent_onprem_configuration
+```
+
+### 3. Verify S3 Buckets
+
+```bash
+# List buckets
+aws s3 ls | grep <customer-name>
+
+# Verify bucket policies
+aws s3api get-bucket-policy --bucket <bucket-name>
+```
+
+### 4. Verify Cross-Account Access
+
+After Minded team confirms configuration:
+
+1. Create a test agent in Minded platform
+2. Deploy to dev environment
+3. Check pod status:
+   ```bash
+   kubectl get pods -n dev
+   ```
+4. Run a test session
+5. Verify data in S3:
+   ```bash
+   aws s3 ls s3://<agent-data-bucket>/
+   ```
+
+## Outputs
+
+After successful deployment, Terraform provides outputs needed by Minded:
+
+```hcl
+{
+  "customer"                         = "your-company-name"
+  "documentdb_endpoint_service_name" = "com.amazonaws.vpce.us-east-1.vpce-svc-xxxxx"
+  "s3_bucket"                        = "minded-agents-data"
+  "repository_url"                   = "123456789012.dkr.ecr.us-east-1.amazonaws.com/company-production-agent"
+  "assume_role_arn"                  = "arn:aws:iam::123456789012:role/.../backend-role"
+  "kms_key_id"                       = "arn:aws:kms:us-east-1:123456789012:key/xxxxx"
+  "environments"                     = ["dev", "stg", "prod"]
+  "k8s_service_account_role_arn"     = "arn:aws:iam::123456789012:role/.../k8s-role"
+}
+```
+
+**Share these outputs with Minded team** via secure channel.
+
+## Next Steps
+
+After deployment is complete:
+
+1. **Share Outputs**: Send the Terraform outputs to Minded team via secure channel
+2. **Wait for Configuration**: Minded team will configure PrivateLink connection
+3. **Create Your First Agent**: Use the Minded platform to build and deploy

package/docs/self-hosting/examples/README.md

@@ -0,0 +1,303 @@
+# AWS Deployment Reference Examples
+
+This directory contains reference configuration files that show what Terraform creates and how components are configured. These files complement the [Terraform-based AWS Deployment](../aws.md).
+
+## Purpose
+
+These examples are for **reference and troubleshooting only**:
+
+- Understand IAM policies and trust relationships
+- Debug access issues
+- Learn about the infrastructure components
+
+**Important**: Do NOT apply these manually. Terraform handles all infrastructure provisioning.
+
+## Files
+
+### IAM Trust Policies
+
+#### `trust-policy-cross-account.json`
+
+Trust policy for the cross-account role that Minded backend assumes.
+
+**Purpose**: Allows Minded backend to:
+
+- Push Docker images to your ECR
+- Deploy agents to your EKS cluster
+- Access your S3 agent data bucket
+
+**Created by**: Terraform (`modules/agent-on-prem/iam_backend_role.tf`)
+
+**Key Points**:
+
+- Uses `sts:AssumeRole`
+- Requires External ID for security
+- Principal is Minded AWS account root (Minded backend assumes this role)
+
+#### `trust-policy-irsa.json`
+
+Trust policy for the IRSA (IAM Roles for Service Accounts) role used by agent pods.
+
+**Purpose**: Allows agent pods to:
+
+- Write logs, screenshots, and files to S3
+
+**Created by**: Terraform (`modules/agent-on-prem/iam_pod_exec_role.tf`)
+
+**Key Points**:
+
+- Uses `sts:AssumeRoleWithWebIdentity` for IRSA
+- Service account name is `onprem-agent` (created by Terraform)
+- One role used across all namespaces (dev, stg, prod)
+
+### IAM Policies
+
+#### `s3-iam-policy.json`
+
+IAM policy attached to the pod execution role (IRSA).
+
+**Created by**: Terraform automatically attaches this to the pod role
+
+**Permissions**:
+
+- Full S3 access to agent data bucket
+- Used by agent pods to store:
+  - Session logs
+  - Browser screenshots
+  - Downloaded/manipulated files
+
+### S3 Configuration
+
+#### `bucket-policy.json`
+
+Example S3 bucket policy (optional - not used by default Terraform).
+
+- Data is organized by agent using prefixes: `/agent-id/session-id/...`
+- Example path:
+  - `/agent-123/session-456/*`
+
+**Note**: The default Terraform deployment uses IAM roles for access control (not bucket policies). This file is provided as a reference if you need to add additional access controls.
+
+Access is controlled by:
+
+- **Minded backend**: Via cross-account IAM role (full bucket access)
+- **Agent pods**: Via IRSA role (full bucket access, organized by prefixes)
+
+#### `lifecycle-policy.json`
+
+S3 lifecycle rules for cost optimization.
+
+**Usage**: Apply after Terraform deployment to manage data retention:
+
+```bash
+aws s3api put-bucket-lifecycle-configuration \
+  --bucket minded-agents-data \
+  --lifecycle-configuration file://lifecycle-policy.json
+```
+
+**Default Rules**:
+
+- Transition to Standard-IA after 30 days
+- Transition to Glacier after 60 days
+- Delete after 90 days
+
+Customize based on your compliance and retention requirements.
+
+## Terraform Deployment (Recommended)
+
+The Terraform configuration automatically creates all necessary IAM roles and policies:
+
+✅ Cross-account role for Minded backend
+✅ IRSA role for agent pods  
+✅ S3 buckets with proper encryption
+✅ EKS cluster and namespaces
+✅ ECR repository
+✅ DocumentDB with PrivateLink
+
+See [AWS Deployment Guide](../aws.md) for complete instructions.
+
+## Common Use Cases
+
+### Troubleshooting Agent Pod S3 Access
+
+If agent pods can't write to S3:
+
+1. Verify pod is using correct service account:
+
+   ```bash
+   kubectl get pod <POD_NAME> -n dev -o yaml | grep serviceAccountName
+   # Should show: onprem-agent
+   ```
+
+2. Check service account has IRSA annotation:
+
+   ```bash
+   kubectl get sa onprem-agent -n dev -o yaml
+   # Should have: eks.amazonaws.com/role-arn annotation
+   ```
+
+3. Verify IAM role has S3 permissions:
+
+   ```bash
+   aws iam get-role-policy --role-name <ROLE_NAME> --policy-name PodExec
+   ```
+
+4. Compare with `s3-iam-policy.json` example
+
+### Troubleshooting Minded Backend Access
+
+If Minded backend can't push images or deploy:
+
+1. Verify cross-account role exists:
+
+   ```bash
+   aws iam get-role --role-name <BACKEND_ROLE_NAME>
+   ```
+
+2. Check trust policy includes Minded account:
+
+   ```bash
+   aws iam get-role --role-name <BACKEND_ROLE_NAME> \
+     --query 'Role.AssumeRolePolicyDocument'
+   ```
+
+3. Compare with `trust-policy-cross-account.json`
+
+4. Verify external ID matches what you provided to Minded
+
+### Custom Lifecycle Policies
+
+To adjust data retention:
+
+1. Copy and modify `lifecycle-policy.json`
+2. Update transition days and storage classes
+3. Apply to your bucket:
+   ```bash
+   aws s3api put-bucket-lifecycle-configuration \
+     --bucket <BUCKET_NAME> \
+     --lifecycle-configuration file://custom-lifecycle.json
+   ```
+
+Example for longer retention:
+
+```json
+{
+  "Rules": [
+    {
+      "Id": "LongTermRetention",
+      "Status": "Enabled",
+      "Transitions": [
+        {
+          "Days": 90,
+          "StorageClass": "STANDARD_IA"
+        },
+        {
+          "Days": 180,
+          "StorageClass": "GLACIER"
+        }
+      ],
+      "Expiration": {
+        "Days": 365
+      }
+    }
+  ]
+}
+```
+
+## What NOT to Do
+
+❌ **Don't manually deploy agents** - Minded backend handles this
+❌ **Don't modify IAM roles created by Terraform** - Will break deployments  
+❌ **Don't change service account name** - Must be `onprem-agent`
+❌ **Don't add LoadBalancer services** - Agents connect outbound via WebSocket
+❌ **Don't configure DocumentDB access from pods** - Access is via Minded backend only
+
+## What You CAN Customize
+
+✅ S3 lifecycle policies (after Terraform deployment)
+✅ CloudWatch alarms and dashboards
+✅ Additional S3 bucket policies (if needed for your tools)
+✅ Resource limits in agent pods (contact Minded support)
+
+## Validation
+
+### Validate JSON Files
+
+```bash
+# Check JSON syntax
+jq . trust-policy-cross-account.json
+jq . trust-policy-irsa.json
+jq . s3-iam-policy.json
+jq . bucket-policy.json
+jq . lifecycle-policy.json
+```
+
+### Test IAM Policy
+
+Use IAM Policy Simulator to test S3 access:
+
+```bash
+aws iam simulate-principal-policy \
+  --policy-source-arn arn:aws:iam::ACCOUNT:role/pod-exec-role \
+  --action-names s3:PutObject s3:GetObject \
+  --resource-arns arn:aws:s3:::bucket-name/test-file.txt
+```
+
+### Verify IRSA Configuration
+
+```bash
+# Check OIDC provider exists
+aws iam list-open-id-connect-providers
+
+# Verify pod identity association
+aws eks list-pod-identity-associations --cluster-name <CLUSTER_NAME>
+```
+
+## Architecture Reminders
+
+### Data Flow
+
+```
+Agent Pod → S3 (direct via IRSA)
+Agent Pod → WebSocket → Minded Backend → DocumentDB (0-retention)
+                                       → S3 (retrieval)
+```
+
+### Access Patterns
+
+**Agent Pods**:
+
+- ✅ Write to S3 (logs, screenshots, files)
+- ✅ Connect to Minded backend via WebSocket
+- ❌ No DocumentDB access
+- ❌ No ElastiCache access
+
+**Minded Backend**:
+
+- ✅ Assume cross-account role
+- ✅ Push to ECR
+- ✅ Deploy to EKS
+- ✅ Read/write S3
+- ✅ Read/write DocumentDB via PrivateLink (0-retention)
+
+## Support
+
+For questions about these examples:
+
+- **Terraform Issues**: https://github.com/minded-ai/self-hosting-terraform/issues
+- **General Support**: support@minded.ai
+- **Documentation**: [AWS Deployment Guide](../aws.md)
+
+## File Summary
+
+| File                              | Purpose                     | Used By        | Applied By        |
+| --------------------------------- | --------------------------- | -------------- | ----------------- |
+| `trust-policy-cross-account.json` | Cross-account role trust    | Minded Backend | Terraform         |
+| `trust-policy-irsa.json`          | Pod IRSA trust policy       | Agent Pods     | Terraform         |
+| `s3-iam-policy.json`              | S3 access permissions       | Agent Pods     | Terraform         |
+| `bucket-policy.json`              | S3 bucket policy (optional) | N/A            | Manual (optional) |
+| `lifecycle-policy.json`           | S3 data retention           | N/A            | Manual (optional) |
+
+---
+
+**Remember**: These are reference examples. Terraform and Minded backend handle all actual deployments automatically.

package/docs/self-hosting/examples/bucket-policy.json

@@ -0,0 +1,19 @@
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Sid": "MindedBackendCrossAccountAccess",
+      "Effect": "Allow",
+      "Principal": {
+        "AWS": "arn:aws:iam::MINDED_AWS_ACCOUNT:root"
+      },
+      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListBucket", "s3:GetBucketLocation"],
+      "Resource": ["arn:aws:s3:::minded-agents-data", "arn:aws:s3:::minded-agents-data/*"],
+      "Condition": {
+        "StringEquals": {
+          "sts:ExternalId": "YOUR_EXTERNAL_ID"
+        }
+      }
+    }
+  ]
+}

package/docs/self-hosting/examples/lifecycle-policy.json

@@ -0,0 +1,35 @@
+{
+  "Rules": [
+    {
+      "Id": "DeleteOldLogs",
+      "Status": "Enabled",
+      "Prefix": "logs/",
+      "Expiration": {
+        "Days": 90
+      }
+    },
+    {
+      "Id": "TransitionOldLogsToIA",
+      "Status": "Enabled",
+      "Prefix": "logs/",
+      "Transitions": [
+        {
+          "Days": 30,
+          "StorageClass": "STANDARD_IA"
+        },
+        {
+          "Days": 60,
+          "StorageClass": "GLACIER"
+        }
+      ]
+    },
+    {
+      "Id": "DeleteOldScreenshots",
+      "Status": "Enabled",
+      "Prefix": "screenshots/",
+      "Expiration": {
+        "Days": 30
+      }
+    }
+  ]
+}

package/docs/self-hosting/examples/s3-iam-policy.json

@@ -0,0 +1,21 @@
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Sid": "S3AgentDataBucketAccess",
+      "Effect": "Allow",
+      "Action": [
+        "s3:PutObject",
+        "s3:PutObjectTagging",
+        "s3:GetObjectAcl",
+        "s3:GetObject",
+        "s3:DeleteObjectVersion",
+        "s3:DeleteObject",
+        "s3:GetBucketLocation",
+        "s3:GetObjectVersion",
+        "s3:List*"
+      ],
+      "Resource": ["arn:aws:s3:::minded-agents-data", "arn:aws:s3:::minded-agents-data/*"]
+    }
+  ]
+}