flow-frame-core 0.1.0

package/dist/README-H100-VM.md

@@ -0,0 +1,192 @@
+# A100/H100 VM Deployment System
+
+This deployment system replaces Cloud Run worker pools with powerful VM instances equipped with NVIDIA A100 or H100 GPUs for high-performance ML workloads.
+
+## Quick Start
+
+1. **Deploy a new H100 worker:**
+   ```bash
+   ./deploy-vm-h100.sh
+   ```
+
+2. **Check status:**
+   ```bash
+   ./manage-vm-h100.sh status
+   ```
+
+3. **Stop when not in use (saves costs):**
+   ```bash
+   ./manage-vm-h100.sh stop
+   ```
+
+## Files
+
+- `deploy-vm-h100.sh` - Main deployment script
+- `manage-vm-h100.sh` - Management and monitoring script
+- `vm-h100.env.template` - Configuration template
+- `deploy-worker-pools.sh` - Original Cloud Run deployment (legacy)
+
+## Configuration
+
+Copy the template and customize:
+```bash
+cp vm-h100.env.template .env
+# Edit .env with your settings
+source .env
+./deploy-vm-h100.sh
+```
+
+### Key Configuration Options
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `MACHINE_TYPE` | `a2-highgpu-1g` | VM type (1g/2g/4g/8g for 1/2/4/8 A100s) |
+| `GPU_COUNT` | `1` | Number of A100 GPUs |
+| `ZONE` | `us-central1-a` | GCP zone (must support A2 instances) |
+| `EXTERNAL_IP` | `ephemeral` | External IP assignment |
+| `GPU_MEMORY_FRACTION` | `0.9` | GPU memory utilization |
+
+## Machine Types & Costs
+
+| Machine Type | GPUs | RAM | Approximate Cost/Hour |
+|--------------|------|-----|----------------------|
+| `a2-highgpu-1g` | 1x A100 | 26GB | ~$15 |
+| `a2-highgpu-2g` | 2x A100 | 52GB | ~$30 |
+| `a2-highgpu-4g` | 4x A100 | 104GB | ~$60 |
+| `a2-highgpu-8g` | 8x A100 | 208GB | ~$120 |
+
+> **💰 Cost Warning:** A100 instances are expensive! Always stop when not in use.
+
+## Management Commands
+
+```bash
+# Deploy new instance
+./manage-vm-h100.sh deploy
+
+# Instance lifecycle
+./manage-vm-h100.sh start
+./manage-vm-h100.sh stop
+./manage-vm-h100.sh restart
+
+# Monitoring
+./manage-vm-h100.sh status
+./manage-vm-h100.sh logs
+./manage-vm-h100.sh gpu
+
+# Maintenance
+./manage-vm-h100.sh update
+./manage-vm-h100.sh ssh
+./manage-vm-h100.sh cleanup
+```
+
+## GPU Monitoring
+
+Monitor GPU usage in real-time:
+```bash
+./manage-vm-h100.sh gpu
+```
+
+Or SSH and use nvidia-smi directly:
+```bash
+./manage-vm-h100.sh ssh
+# Then on the VM:
+watch nvidia-smi
+```
+
+## Differences from Cloud Run
+
+| Feature | Cloud Run | A100 VMs |
+|---------|-----------|----------|
+| **GPUs** | L4 (24GB) | A100 (40GB) |
+| **Max GPUs** | 1 | 8 |
+| **Memory** | 32GB max | 208GB max |
+| **Scaling** | Automatic | Manual |
+| **Cost Model** | Per-request | Per-hour |
+| **Startup Time** | ~30s | ~2-5min |
+| **Persistence** | Stateless | Persistent |
+| **Networking** | Managed | VM-based |
+
+## Zones Supporting A2 Instances
+
+A100 GPUs are available in these zones:
+- `us-central1-a` (recommended)
+- `us-central1-a`
+- `us-east4-a`
+- `us-east4-c`
+- `us-west4-a`
+- `us-west4-b`
+- `europe-west4-a`
+- `asia-southeast1-c`
+
+## Troubleshooting
+
+### Instance won't start
+- Check quota limits for A2 instances in your project
+- Verify the zone supports A2 instances
+- Ensure GPU quotas are sufficient
+
+### Container fails to access GPUs
+- Check NVIDIA drivers: `./manage-vm-h100.sh ssh` → `nvidia-smi`
+- Verify container runtime: `docker info | grep nvidia`
+- Check container logs: `./manage-vm-h100.sh logs`
+
+### High costs
+- Always stop instances when not in use: `./manage-vm-h100.sh stop`
+- Consider smaller machine types for development
+- Use preemptible instances for fault-tolerant workloads
+
+### Performance issues
+- Adjust `GPU_MEMORY_FRACTION` in configuration
+- Monitor GPU utilization: `./manage-vm-h100.sh gpu`
+- Check for CPU bottlenecks in container
+
+## Migration from Cloud Run
+
+To migrate from the existing Cloud Run worker pools:
+
+1. **Deploy A100 VM:**
+   ```bash
+   ./deploy-vm-h100.sh
+   ```
+
+2. **Update application configuration** to point to the new VM endpoint
+
+3. **Scale down Cloud Run workers:**
+   ```bash
+   WORKER_COUNT=0 ./deploy-worker-pools.sh
+   ```
+
+4. **Clean up Cloud Run services** (optional):
+   ```bash
+   gcloud run services delete midjourney-worker-pool-1 --region=us-central1
+   ```
+
+## Security Considerations
+
+- VMs have full cloud-platform scope by default
+- Firewall rules allow HTTP/HTTPS from anywhere
+- Consider restricting network access for production
+- Ensure container images are from trusted sources
+- Monitor for unauthorized usage due to high costs
+
+## Best Practices
+
+1. **Cost Management:**
+   - Stop instances when not training/inferencing
+   - Use monitoring alerts for unexpected usage
+   - Consider scheduled start/stop for batch workloads
+
+2. **Performance:**
+   - Match batch size to GPU memory capacity
+   - Use GPU memory profiling to optimize utilization
+   - Consider multi-GPU parallelization strategies
+
+3. **Reliability:**
+   - Regular snapshots of boot disks with models/data
+   - Health check endpoints in containers
+   - Automated restart policies for long-running workloads
+
+4. **Development:**
+   - Use smaller instances (a2-highgpu-1g) for development
+   - Test with synthetic data before scaling up
+   - Profile GPU usage before moving to production

package/dist/README-worker-pools.md

@@ -0,0 +1,231 @@
+# Worker Pool Deployment Scripts
+
+This directory contains scripts for deploying and managing Cloud Run worker pools for the midjourney service.
+
+## Scripts
+
+### 1. `deploy-worker-pools.sh`
+Deploys multiple Cloud Run services as worker pools for parallel job processing.
+
+### 2. `manage-worker-pools.sh` 
+Manages existing worker pool deployments (scale, restart, monitor, etc.).
+
+## Quick Start
+
+### Deploy Worker Pools
+
+```bash
+# Deploy 3 workers with default settings
+./deploy-worker-pools.sh
+
+# Deploy 5 workers with custom configuration
+WORKER_COUNT=5 CPU=8 MEMORY=32Gi ./deploy-worker-pools.sh
+
+# Deploy workers with NVIDIA L4 GPUs (Cloud Run supported, more powerful than T4)
+WORKER_COUNT=3 CPU=8 MEMORY=32Gi GPU_TYPE=nvidia-l4 GPU_COUNT=1 ./deploy-worker-pools.sh
+
+# Deploy workers with custom environment variables
+BASE_ENV_VARS="ENABLE_GPU=true,MAX_CONCURRENT_JOBS=2" WORKER_COUNT=2 ./deploy-worker-pools.sh
+
+# Use npm scripts for common configurations
+npm run workers:deploy:gpu-l4    # L4 GPU workers (8 CPU, 32GB RAM) - RECOMMENDED
+npm run workers:deploy:gpu-a100  # A100 GPU workers (most powerful)
+npm run workers:deploy:gpu-h100  # H100 GPU workers (latest/fastest)
+```
+
+### Manage Worker Pools
+
+```bash
+# List all worker services
+./manage-worker-pools.sh list
+
+# Check health status
+./manage-worker-pools.sh health
+
+# View detailed status
+./manage-worker-pools.sh status
+
+# Show logs for worker 1
+./manage-worker-pools.sh logs 1
+
+# Scale to 5 workers
+./manage-worker-pools.sh scale 5
+
+# Restart specific worker
+./manage-worker-pools.sh restart 2
+
+# Restart all workers
+./manage-worker-pools.sh restart-all
+```
+
+## Configuration
+
+### Environment Variables
+
+#### Deploy Script (`deploy-worker-pools.sh`)
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `WORKER_COUNT` | `3` | Number of worker instances to deploy |
+| `CPU` | `4` | CPU allocation per worker |
+| `MEMORY` | `16Gi` | Memory allocation per worker |
+| `GPU_TYPE` | `""` | GPU type (nvidia-l4, nvidia-a100-80gb, nvidia-h100-80gb) |
+| `GPU_COUNT` | `""` | Number of GPUs per worker (1 for Cloud Run) |
+| `MIN_INSTANCES` | `1` | Minimum instances (keeps workers warm) |
+| `MAX_INSTANCES` | `1` | Maximum instances per worker service |
+| `TIMEOUT` | `3600` | Request timeout in seconds (1 hour) |
+| `WORKER_MODE` | `true` | Flag to indicate worker mode |
+| `ENABLE_TRAINING` | `true` | Enable ML training capabilities |
+| `BASE_ENV_VARS` | `""` | Additional environment variables |
+
+#### Examples
+
+```bash
+# Deploy high-memory workers for training
+CPU=8 MEMORY=32Gi ENABLE_TRAINING=true WORKER_COUNT=2 ./deploy-worker-pools.sh
+
+# Deploy GPU workers with L4 (more powerful than T4, Cloud Run supported)
+CPU=8 MEMORY=32Gi GPU_TYPE=nvidia-l4 GPU_COUNT=1 WORKER_COUNT=3 ./deploy-worker-pools.sh
+
+# Deploy processing-only workers
+ENABLE_TRAINING=false CPU=2 MEMORY=8Gi WORKER_COUNT=5 ./deploy-worker-pools.sh
+
+# Deploy high-end A100 GPU workers
+CPU=8 MEMORY=32Gi GPU_TYPE=nvidia-a100-80gb GPU_COUNT=1 WORKER_COUNT=1 ./deploy-worker-pools.sh
+
+# Deploy with custom environment
+BASE_ENV_VARS="WORKER_TYPE=training,MAX_JOBS=1,GPU_ENABLED=true" ./deploy-worker-pools.sh
+
+# Use npm scripts for quick deployment
+npm run workers:deploy:gpu-l4        # L4 GPU workers (RECOMMENDED for your needs)
+npm run workers:deploy:gpu-a100      # A100 GPU workers (most powerful)
+npm run workers:deploy:gpu-h100      # H100 GPU workers (latest)
+```
+
+## Worker Pool Architecture
+
+Each worker pool deployment creates:
+
+- **Individual Cloud Run Services**: `midjourney-worker-1`, `midjourney-worker-2`, etc.
+- **Isolated Processing**: Each worker handles jobs independently
+- **Auto-scaling**: Workers can scale to 0 when idle (configurable)
+- **Resource Isolation**: Each worker has dedicated CPU/memory allocation
+- **Health Monitoring**: Built-in health checks and status monitoring
+
+## Monitoring and Troubleshooting
+
+### View Worker Status
+```bash
+./manage-worker-pools.sh status
+```
+
+### Check Logs
+```bash
+# View logs for specific worker
+./manage-worker-pools.sh logs 1
+
+# View logs with gcloud directly
+gcloud logs read "resource.type=cloud_run_revision AND resource.labels.service_name=midjourney-worker-1" --limit=50
+```
+
+### Health Checks
+```bash
+./manage-worker-pools.sh health
+```
+
+### Scaling Operations
+```bash
+# Scale up
+./manage-worker-pools.sh scale 10
+
+# Scale down
+./manage-worker-pools.sh scale 2
+
+# Scale to zero (stop all workers)
+./manage-worker-pools.sh scale 0
+```
+
+## Advanced Configuration
+
+### Pool-Specific Environment Variables
+
+You can configure different environment variables for specific worker pools:
+
+```bash
+POOL_ENV_VARS='{"pool-1":"WORKER_TYPE=training","pool-2":"WORKER_TYPE=processing"}' \
+WORKER_COUNT=2 \
+./deploy-worker-pools.sh
+```
+
+### Custom Worker Naming
+
+```bash
+WORKER_SUFFIX="-gpu" BASE_SERVICE="ml-worker" ./deploy-worker-pools.sh
+# Creates: ml-worker-gpu-1, ml-worker-gpu-2, etc.
+```
+
+### Using Docker Hub Images
+
+```bash
+USE_DOCKERHUB=true DOCKERHUB_IMAGE="your-registry/worker:latest" ./deploy-worker-pools.sh
+```
+
+## Job Processing Integration
+
+Workers automatically:
+
+1. **Connect to Firestore** for job coordination
+2. **Claim available subjobs** using atomic transactions
+3. **Process jobs** based on configuration (training/processing)
+4. **Update job status** in real-time
+5. **Handle failures** with proper error reporting
+
+The worker system integrates with the existing job queue architecture in `capture_training_data_endpoints.js`.
+
+## Cleanup
+
+### Delete Specific Worker
+```bash
+./manage-worker-pools.sh delete 3
+```
+
+### Delete All Workers
+```bash
+./manage-worker-pools.sh delete-all
+```
+
+## Cost Optimization
+
+- **Min Instances**: Set to 0 for cost savings (workers scale to zero when idle)
+- **Resource Allocation**: Adjust CPU/memory based on workload requirements
+- **Worker Count**: Scale based on job queue depth
+- **Timeouts**: Optimize timeout values for your job processing times
+
+## Security
+
+- Workers can be deployed with or without authentication
+- Use service accounts for Firestore/Storage access
+- Environment variables can include sensitive configuration
+- Network policies can restrict worker communication
+
+## Troubleshooting
+
+### Common Issues
+
+1. **Workers not claiming jobs**: Check Firestore permissions and worker configuration
+2. **High memory usage**: Increase memory allocation or reduce concurrency
+3. **Timeouts**: Increase timeout values for long-running jobs
+4. **Image build failures**: Check Docker configuration and registry permissions
+
+### Debug Commands
+
+```bash
+# Check worker configuration
+gcloud run services describe midjourney-worker-1 --region=us-central1
+
+# View recent logs
+./manage-worker-pools.sh logs 1
+
+# Test worker health
+curl -s https://midjourney-worker-1-xxx.run.app/health
+```

package/dist/README.md

@@ -0,0 +1,8 @@
+# Check VM status
+gcloud compute instances list --project=flowframe-auto
+
+# Monitor startup logs
+gcloud compute instances get-serial-port-output instance-20250913-040737 --zone=us-central1-f --project=flowframe-auto
+
+# Test GPU access
+gcloud compute ssh instance-20250913-040737 --zone=us-central1-f --project=flowframe-auto --command='nvidia-smi'