agentic-flow 1.8.15 → 1.9.0

package/agentic-flow/examples/crispr-cas13-pipeline/docs/architecture-diagrams/c4-component.md

@@ -0,0 +1,240 @@
+# C4 Component Diagram - API Gateway & Processing Services
+
+## API Gateway Components
+
+This diagram shows the internal structure of the API Gateway container.
+
+```mermaid
+C4Component
+    title Component Diagram - API Gateway (Axum/Rust)
+
+    Container_Boundary(api_gateway, "API Gateway") {
+        Component(router, "Router", "Axum Router", "Routes HTTP requests to handlers")
+        Component(auth_middleware, "Auth Middleware", "Tower Layer", "JWT validation, permission checks")
+        Component(rate_limiter, "Rate Limiter", "Tower Layer", "Redis-backed sliding window")
+        Component(logger, "Request Logger", "Tower Layer", "Structured logging with tracing")
+
+        Component(auth_handler, "Auth Handler", "Axum Handler", "Login, logout, token refresh")
+        Component(experiment_handler, "Experiment Handler", "Axum Handler", "CRUD for experiments")
+        Component(sample_handler, "Sample Handler", "Axum Handler", "Upload FASTQ files")
+        Component(job_handler, "Job Handler", "Axum Handler", "Submit and monitor jobs")
+        Component(result_handler, "Result Handler", "Axum Handler", "Query analysis results")
+
+        Component(db_service, "Database Service", "SQLx", "PostgreSQL connection pool")
+        Component(cache_service, "Cache Service", "redis-rs", "Redis client")
+        Component(kafka_service, "Kafka Service", "rdkafka", "Message producer")
+        Component(storage_service, "Storage Service", "rusoto_s3", "MinIO S3 client")
+    }
+
+    ContainerDb(postgres, "PostgreSQL", "Database")
+    ContainerDb(redis, "Redis", "Cache")
+    ContainerQueue(kafka, "Kafka", "Message Queue")
+    ContainerDb(minio, "MinIO", "Object Storage")
+
+    Rel(router, auth_middleware, "Applies", "Layer")
+    Rel(router, rate_limiter, "Applies", "Layer")
+    Rel(router, logger, "Applies", "Layer")
+
+    Rel(auth_middleware, redis, "Verify session", "TCP")
+
+    Rel(router, auth_handler, "Routes /auth/*")
+    Rel(router, experiment_handler, "Routes /experiments/*")
+    Rel(router, sample_handler, "Routes /samples/*")
+    Rel(router, job_handler, "Routes /jobs/*")
+    Rel(router, result_handler, "Routes /results/*")
+
+    Rel(auth_handler, db_service, "Uses")
+    Rel(experiment_handler, db_service, "Uses")
+    Rel(sample_handler, storage_service, "Uses")
+    Rel(job_handler, kafka_service, "Uses")
+    Rel(result_handler, db_service, "Uses")
+
+    Rel(db_service, postgres, "Queries", "TCP/5432")
+    Rel(cache_service, redis, "Get/Set", "TCP/6379")
+    Rel(kafka_service, kafka, "Publish", "TCP/9092")
+    Rel(storage_service, minio, "Upload/Download", "HTTPS/9000")
+
+    UpdateLayoutConfig($c4ShapeInRow="4", $c4BoundaryInRow="1")
+```
+
+## API Gateway Component Details
+
+### 1. Middleware Stack
+
+**Execution Order** (outermost to innermost):
+1. **Request Logger**: Logs all incoming requests (method, path, headers)
+2. **Rate Limiter**: Checks Redis for rate limit, returns 429 if exceeded
+3. **Auth Middleware**: Validates JWT token, extracts user claims
+4. **Router**: Routes request to appropriate handler
+
+**Tower Middleware Example**:
+```rust
+let app = Router::new()
+    .nest("/api/v1", api_routes)
+    .layer(
+        ServiceBuilder::new()
+            .layer(TraceLayer::new_for_http())  // Request logger
+            .layer(middleware::from_fn(rate_limit_middleware))
+            .layer(middleware::from_fn(auth_middleware))
+    );
+```
+
+### 2. Route Handlers
+
+**Auth Handler** (`/api/v1/auth/*`)
+- `POST /login`: Username/password → JWT token
+- `POST /logout`: Invalidate session
+- `POST /refresh`: Refresh token → New JWT
+- `GET /verify`: Validate current token
+
+**Experiment Handler** (`/api/v1/experiments/*`)
+- `GET /`: List experiments (filtered by user)
+- `POST /`: Create new experiment
+- `GET /:id`: Get experiment details
+- `PUT /:id`: Update experiment
+- `DELETE /:id`: Delete experiment
+
+**Sample Handler** (`/api/v1/samples/*`)
+- `POST /upload`: Upload FASTQ files to MinIO
+- `GET /:id`: Get sample metadata
+- `DELETE /:id`: Delete sample
+
+**Job Handler** (`/api/v1/jobs/*`)
+- `POST /`: Submit analysis job to Kafka
+- `GET /:id`: Get job status
+- `GET /:id/logs`: Stream job logs
+- `DELETE /:id`: Cancel running job
+
+**Result Handler** (`/api/v1/results/*`)
+- `GET /off-targets/:job_id`: Query off-target predictions
+- `GET /diff-expr/:job_id`: Query differential expression results
+- `GET /immune/:job_id`: Query immune response signatures
+- `GET /export/:job_id`: Generate CSV/JSON export
+
+### 3. Service Layer
+
+**Database Service**
+- **Connection Pool**: 100 max connections (SQLx)
+- **Query Builder**: Type-safe queries with `sqlx::query!` macro
+- **Transactions**: Automatic rollback on error
+- **Migrations**: Versioned schema migrations
+
+**Cache Service**
+- **Client**: redis-rs with connection pooling
+- **Patterns**:
+  - Session storage (`session:{token}`)
+  - Rate limiting (`rate_limit:{user}:{minute}`)
+  - Job status (`job:{id}`)
+- **TTL**: Configurable per key (default: 5 minutes)
+
+**Kafka Service**
+- **Producer**: rdkafka FutureProducer (async)
+- **Serialization**: JSON with `serde_json`
+- **Topics**: `jobs.alignment`, `jobs.off_target`, `jobs.diff_expr`, `jobs.immune`
+- **Error Handling**: Retry 3 times with exponential backoff
+
+**Storage Service**
+- **Client**: rusoto_s3 (AWS SDK for Rust)
+- **Buckets**: `crispr-cas13` (main), `crispr-cas13-backups`
+- **Pre-signed URLs**: 1-hour TTL for downloads
+- **Multipart Upload**: For files >5GB
+
+---
+
+## Processing Service Components (Alignment Service)
+
+```mermaid
+C4Component
+    title Component Diagram - Alignment Service (Python/Bowtie2)
+
+    Container_Boundary(alignment_service, "Alignment Service") {
+        Component(kafka_consumer, "Kafka Consumer", "kafka-python", "Consumes alignment jobs")
+        Component(file_downloader, "File Downloader", "boto3", "Downloads FASTQ from MinIO")
+        Component(bowtie2_wrapper, "Bowtie2 Wrapper", "subprocess", "Runs Bowtie2 alignment")
+        Component(samtools_wrapper, "Samtools Wrapper", "subprocess", "BAM conversion & indexing")
+        Component(qc_module, "QC Module", "FastQC/MultiQC", "Quality control reports")
+        Component(file_uploader, "File Uploader", "boto3", "Uploads BAM to MinIO")
+        Component(db_updater, "DB Updater", "psycopg2", "Updates job status in PostgreSQL")
+        Component(metrics_collector, "Metrics Collector", "prometheus_client", "Exposes metrics")
+    }
+
+    ContainerQueue(kafka, "Kafka", "Message Queue")
+    ContainerDb(minio, "MinIO", "Object Storage")
+    ContainerDb(postgres, "PostgreSQL", "Database")
+
+    Rel(kafka, kafka_consumer, "Delivers jobs", "TCP/9092")
+    Rel(kafka_consumer, file_downloader, "Triggers")
+    Rel(file_downloader, minio, "Download FASTQ", "HTTPS/9000")
+
+    Rel(kafka_consumer, bowtie2_wrapper, "Executes")
+    Rel(bowtie2_wrapper, samtools_wrapper, "Pipes SAM")
+    Rel(samtools_wrapper, qc_module, "Generates stats")
+
+    Rel(qc_module, file_uploader, "Upload BAM + reports")
+    Rel(file_uploader, minio, "Upload", "HTTPS/9000")
+
+    Rel(kafka_consumer, db_updater, "Update status")
+    Rel(db_updater, postgres, "INSERT/UPDATE", "TCP/5432")
+
+    Rel(kafka_consumer, metrics_collector, "Record metrics")
+
+    UpdateLayoutConfig($c4ShapeInRow="4", $c4BoundaryInRow="1")
+```
+
+## Alignment Service Workflow
+
+1. **Consume Job**: Kafka consumer receives alignment job message
+2. **Download FASTQ**: Download R1/R2 FASTQ files from MinIO to `/tmp`
+3. **Run Bowtie2**: Execute Bowtie2 alignment (8 threads)
+   ```bash
+   bowtie2 -x ref_genome -1 R1.fastq.gz -2 R2.fastq.gz -p 8 --very-sensitive
+   ```
+4. **Convert to BAM**: Pipe SAM output to samtools
+   ```bash
+   samtools view -bS - | samtools sort -o sorted.bam
+   samtools index sorted.bam
+   ```
+5. **Quality Control**: Run FastQC on BAM file
+6. **Upload Results**: Upload BAM, BAM index, QC report to MinIO
+7. **Update Database**: Mark job as completed in PostgreSQL
+8. **Publish Event**: Send completion message to Kafka (`jobs.completed`)
+9. **Cleanup**: Delete temporary files from `/tmp`
+
+## Error Handling Strategy
+
+**Transient Errors** (network failures, timeouts):
+- **Retry**: 3 attempts with exponential backoff (1s, 2s, 4s)
+- **Dead Letter Queue**: After 3 failures, move to `jobs.failed` topic
+
+**Permanent Errors** (invalid input, out of memory):
+- **Immediate Failure**: Mark job as failed in database
+- **Notification**: Send alert to user via email/WebSocket
+- **Logging**: Store detailed error message in PostgreSQL
+
+**Resource Exhaustion**:
+- **Graceful Degradation**: If disk space <10%, pause processing
+- **Auto-Recovery**: Resume when space is freed
+
+---
+
+## Component Communication Patterns
+
+### Synchronous (Request-Response)
+- **API Gateway → PostgreSQL**: Direct database queries (SELECT, INSERT, UPDATE)
+- **API Gateway → Redis**: Cache lookups (GET, SET)
+- **API Gateway → MinIO**: File operations (PUT, GET)
+
+### Asynchronous (Event-Driven)
+- **API Gateway → Kafka**: Job submission (fire-and-forget)
+- **Kafka → Workers**: Job consumption (at-least-once delivery)
+- **Workers → Kafka**: Status updates (completion/failure events)
+
+### Streaming
+- **WebSocket Server → Clients**: Real-time job status updates
+- **Prometheus → Grafana**: Continuous metrics streaming
+
+---
+
+**Diagram Version**: 1.0
+**Last Updated**: 2025-10-12
+**See Also**: [Sequence Diagrams](./sequence-diagrams.md)

package/agentic-flow/examples/crispr-cas13-pipeline/docs/architecture-diagrams/c4-container.md

@@ -0,0 +1,223 @@
+# C4 Container Diagram - CRISPR-Cas13 Pipeline
+
+## Container Architecture
+
+This diagram shows the high-level technology choices and how containers communicate.
+
+```mermaid
+C4Container
+    title Container Diagram - CRISPR-Cas13 Pipeline
+
+    Person(user, "User", "Researcher or Bioinformatician")
+
+    Container_Boundary(frontend, "Client Applications") {
+        Container(web_app, "Web Application", "React/TypeScript", "Interactive UI for experiment management")
+        Container(cli_tool, "CLI Tool", "Python/Rust", "Command-line interface for power users")
+    }
+
+    Container_Boundary(api_layer, "API Layer") {
+        Container(nginx, "Nginx Ingress", "Reverse Proxy", "Load balancing, TLS termination")
+        Container(api_gateway, "API Gateway", "Axum (Rust)", "RESTful API, authentication, rate limiting")
+        Container(websocket_server, "WebSocket Server", "Axum (Rust)", "Real-time job status updates")
+    }
+
+    Container_Boundary(processing_layer, "Processing Layer") {
+        Container(alignment_service, "Alignment Service", "Python/Bowtie2", "Read alignment to genome")
+        Container(off_target_service, "Off-Target Predictor", "Python/PyTorch", "ML-based off-target prediction")
+        Container(diff_expr_service, "Differential Expression", "R/DESeq2", "Gene expression analysis")
+        Container(immune_service, "Immune Response Analyzer", "Python/R", "Signature enrichment analysis")
+    }
+
+    Container_Boundary(data_layer, "Data Layer") {
+        ContainerDb(postgres, "PostgreSQL", "Relational DB", "Experiments, samples, results")
+        ContainerDb(mongodb, "MongoDB", "Document DB", "Genomic annotations, gene ontology")
+        ContainerDb(redis, "Redis", "Cache", "Sessions, job status, rate limiting")
+        ContainerDb(minio, "MinIO", "Object Storage", "FASTQ, BAM, VCF files")
+    }
+
+    Container_Boundary(messaging, "Message Queue") {
+        ContainerQueue(kafka, "Apache Kafka", "Message Broker", "Job orchestration, event streaming")
+    }
+
+    Container_Boundary(monitoring, "Monitoring") {
+        Container(prometheus, "Prometheus", "Metrics", "Time-series metrics collection")
+        Container(grafana, "Grafana", "Dashboards", "Visualization and alerting")
+        Container(jaeger, "Jaeger", "Tracing", "Distributed request tracing")
+    }
+
+    Rel(user, web_app, "Uses", "HTTPS")
+    Rel(user, cli_tool, "Uses", "HTTPS")
+
+    Rel(web_app, nginx, "API requests", "HTTPS/443")
+    Rel(cli_tool, nginx, "API requests", "HTTPS/443")
+    Rel(nginx, api_gateway, "Routes", "HTTP/2")
+    Rel(nginx, websocket_server, "WebSocket", "WSS")
+
+    Rel(api_gateway, postgres, "Reads/Writes", "TCP/5432")
+    Rel(api_gateway, redis, "Cache", "TCP/6379")
+    Rel(api_gateway, kafka, "Publishes jobs", "TCP/9092")
+    Rel(api_gateway, minio, "Upload/Download", "HTTPS/9000")
+
+    Rel(websocket_server, kafka, "Subscribes to updates", "TCP/9092")
+    Rel(websocket_server, redis, "Job status", "TCP/6379")
+
+    Rel(kafka, alignment_service, "Consumes", "TCP/9092")
+    Rel(kafka, off_target_service, "Consumes", "TCP/9092")
+    Rel(kafka, diff_expr_service, "Consumes", "TCP/9092")
+    Rel(kafka, immune_service, "Consumes", "TCP/9092")
+
+    Rel(alignment_service, minio, "Read FASTQ, Write BAM", "HTTPS/9000")
+    Rel(alignment_service, postgres, "Update job status", "TCP/5432")
+    Rel(alignment_service, kafka, "Publish completion", "TCP/9092")
+
+    Rel(off_target_service, minio, "Read BAM", "HTTPS/9000")
+    Rel(off_target_service, postgres, "Write off-targets", "TCP/5432")
+    Rel(off_target_service, mongodb, "Query genome", "TCP/27017")
+
+    Rel(diff_expr_service, minio, "Read BAM", "HTTPS/9000")
+    Rel(diff_expr_service, postgres, "Write results", "TCP/5432")
+
+    Rel(immune_service, postgres, "Read diff expr", "TCP/5432")
+    Rel(immune_service, mongodb, "Query signatures", "TCP/27017")
+
+    Rel(api_gateway, prometheus, "Metrics", "HTTP/9090")
+    Rel(alignment_service, prometheus, "Metrics", "HTTP/9090")
+    Rel(prometheus, grafana, "Queries", "HTTP/3000")
+    Rel(api_gateway, jaeger, "Traces", "UDP/6831")
+
+    UpdateLayoutConfig($c4ShapeInRow="3", $c4BoundaryInRow="2")
+```
+
+## Container Communication Matrix
+
+| Source Container | Target Container | Protocol | Port | Purpose | Security |
+|------------------|------------------|----------|------|---------|----------|
+| Web App → Nginx | API Gateway | HTTPS | 443 | REST API calls | TLS 1.3, JWT |
+| API Gateway → PostgreSQL | Database | TCP | 5432 | CRUD operations | TLS, user/pass |
+| API Gateway → Redis | Cache | TCP | 6379 | Session storage | TLS, password |
+| API Gateway → Kafka | Message Queue | TCP | 9092 | Job submission | SASL/SSL |
+| API Gateway → MinIO | Object Storage | HTTPS | 9000 | File upload/download | IAM keys |
+| Kafka → Workers | Processing | TCP | 9092 | Job distribution | SASL/SSL |
+| Workers → MinIO | Object Storage | HTTPS | 9000 | Read/Write files | IAM keys |
+| Workers → PostgreSQL | Database | TCP | 5432 | Update results | TLS, user/pass |
+| Workers → MongoDB | Document DB | TCP | 27017 | Query annotations | TLS, user/pass |
+| All → Prometheus | Monitoring | HTTP | 9090 | Metrics scraping | Service account |
+
+## Container Technologies
+
+### Frontend Layer
+
+**Web Application**
+- **Framework**: React 18 + TypeScript 5.3
+- **Build Tool**: Vite 5.0
+- **State Management**: Zustand
+- **Deployment**: Nginx static hosting
+- **CDN**: CloudFlare
+
+**CLI Tool**
+- **Language**: Python 3.11 (alternative: Rust)
+- **HTTP Client**: `requests` / `httpx`
+- **Distribution**: PyPI package
+
+### API Layer
+
+**API Gateway**
+- **Framework**: Axum 0.7 (Rust)
+- **Async Runtime**: Tokio 1.35
+- **Middleware**: Tower
+- **Serialization**: Serde
+- **Database Client**: SQLx (PostgreSQL)
+- **Cache Client**: redis-rs
+- **Message Queue**: rdkafka
+- **Deployment**: Docker container on Kubernetes
+
+**WebSocket Server**
+- **Framework**: Axum WebSocket support
+- **Broadcasting**: Tokio broadcast channels
+- **Deployment**: Shared pods with API Gateway
+
+### Processing Layer
+
+**Alignment Service**
+- **Language**: Python 3.11
+- **Bioinformatics**: Bioconda (Bowtie2, Samtools)
+- **Message Queue**: kafka-python
+- **Deployment**: Docker container with Bioconda
+
+**Off-Target Service**
+- **Language**: Python 3.11
+- **ML Framework**: PyTorch 2.1
+- **Genomics**: Biopython
+- **Deployment**: Docker container (GPU-enabled optional)
+
+**Differential Expression Service**
+- **Language**: R 4.3
+- **Framework**: Bioconductor 3.18
+- **Packages**: DESeq2, edgeR, ggplot2
+- **Deployment**: Docker container with R/Bioconductor
+
+**Immune Response Service**
+- **Language**: Python 3.11 + R 4.3 (hybrid)
+- **Packages**: GSEA, MSigDB, ImmuneSigDB
+- **Deployment**: Docker container
+
+### Data Layer
+
+**PostgreSQL**
+- **Version**: 16
+- **Configuration**: Primary + 3 read replicas
+- **Connection Pooling**: PgBouncer
+- **Backup**: Automated daily snapshots
+
+**MongoDB**
+- **Version**: 7.0
+- **Sharding**: 3 shards × 3 replicas
+- **Indexes**: Compound indexes on chromosome + position
+
+**Redis**
+- **Version**: 7.2
+- **Mode**: Cluster (3 masters + 3 replicas)
+- **Persistence**: AOF + RDB snapshots
+
+**MinIO**
+- **Mode**: Distributed (4 nodes × 4 drives)
+- **Erasure Coding**: EC:4 (4 data + 4 parity)
+- **Replication**: Cross-region replication
+
+### Message Queue
+
+**Apache Kafka**
+- **Version**: 3.6
+- **Brokers**: 3 brokers (HA)
+- **Replication**: Factor 3
+- **Partitions**: 10 per topic
+
+### Monitoring
+
+**Prometheus**
+- **Retention**: 30 days
+- **Scrape Interval**: 15 seconds
+- **Storage**: 500GB SSD
+
+**Grafana**
+- **Dashboards**: 5 pre-built dashboards
+- **Alerting**: Prometheus Alertmanager
+
+**Jaeger**
+- **Backend**: Elasticsearch
+- **Retention**: 7 days
+- **Sampling**: 10% of requests
+
+## Deployment Model
+
+All containers run on **Kubernetes 1.28** with:
+- **Namespace Isolation**: `crispr-production`, `crispr-staging`, `crispr-dev`
+- **Resource Quotas**: Per-namespace CPU/memory limits
+- **Network Policies**: Restrict inter-service communication
+- **Ingress Controller**: Nginx Ingress with Let's Encrypt
+
+---
+
+**Diagram Version**: 1.0
+**Last Updated**: 2025-10-12
+**Next Level**: [C4 Component Diagram](./c4-component.md)

package/agentic-flow/examples/crispr-cas13-pipeline/docs/architecture-diagrams/c4-context.md

@@ -0,0 +1,86 @@
+# C4 Context Diagram - CRISPR-Cas13 Pipeline
+
+## System Context
+
+This diagram shows the CRISPR-Cas13 Analysis Pipeline in the context of external users and systems.
+
+```mermaid
+C4Context
+    title System Context Diagram - CRISPR-Cas13 Analysis Pipeline
+
+    Person(researcher, "Research Scientist", "Designs experiments, analyzes results")
+    Person(bioinformatician, "Bioinformatician", "Configures pipelines, validates results")
+    Person(admin, "System Administrator", "Monitors health, manages resources")
+
+    System(crispr_pipeline, "CRISPR-Cas13 Pipeline", "Analyzes off-target effects and immune responses from CRISPR-Cas13 experiments")
+
+    System_Ext(ncbi, "NCBI Database", "Reference genomes and annotations")
+    System_Ext(ensembl, "Ensembl Genome Browser", "Gene annotations and orthologs")
+    System_Ext(immunesigdb, "ImmuneSigDB", "Immune response gene signatures")
+    System_Ext(hpc, "HPC Cluster", "High-performance computing resources")
+
+    Rel(researcher, crispr_pipeline, "Submits samples, views results", "HTTPS")
+    Rel(bioinformatician, crispr_pipeline, "Configures pipelines, exports data", "HTTPS/CLI")
+    Rel(admin, crispr_pipeline, "Monitors system, manages users", "HTTPS/SSH")
+
+    Rel(crispr_pipeline, ncbi, "Downloads reference genomes", "FTP/HTTPS")
+    Rel(crispr_pipeline, ensembl, "Queries gene annotations", "REST API")
+    Rel(crispr_pipeline, immunesigdb, "Retrieves signature data", "REST API")
+    Rel(crispr_pipeline, hpc, "Offloads compute-intensive jobs", "SSH/Slurm")
+
+    UpdateLayoutConfig($c4ShapeInRow="3", $c4BoundaryInRow="1")
+```
+
+## External Systems Integration
+
+### 1. NCBI Database
+- **Purpose**: Download reference genomes (e.g., Macaca mulatta rheMac10)
+- **Protocol**: FTP, HTTPS
+- **Frequency**: Monthly (genome updates)
+- **Authentication**: Public access, API key for higher rate limits
+
+### 2. Ensembl Genome Browser
+- **Purpose**: Gene annotations, transcript variants, orthologs
+- **Protocol**: REST API (`https://rest.ensembl.org/`)
+- **Frequency**: Weekly updates
+- **Rate Limit**: 15 requests/sec
+
+### 3. ImmuneSigDB
+- **Purpose**: Curated immune response gene signatures
+- **Protocol**: REST API, bulk download
+- **Frequency**: Quarterly updates
+- **Data Format**: GMT, JSON
+
+### 4. HPC Cluster (Optional)
+- **Purpose**: Offload large-scale alignment jobs
+- **Protocol**: SSH, Slurm job scheduler
+- **Integration**: Job submission via `sbatch`, result retrieval via `scp`
+- **Use Case**: Experiments with >50 samples
+
+## User Roles & Responsibilities
+
+| Role | Responsibilities | Access Level |
+|------|-----------------|--------------|
+| **Research Scientist** | - Design experiments<br/>- Submit samples<br/>- Interpret results | Read/Write own experiments |
+| **Bioinformatician** | - Configure analysis pipelines<br/>- Validate results<br/>- Export raw data | Read all, Write pipelines |
+| **System Administrator** | - Monitor system health<br/>- Manage users and permissions<br/>- Perform backups | Full system access |
+
+## Security Boundaries
+
+- **External Firewall**: All external traffic goes through firewall (HTTPS only)
+- **API Gateway**: Authentication layer (OAuth2/JWT)
+- **Internal Network**: Service-to-service communication uses mTLS
+- **Data Encryption**: All data encrypted at rest (AES-256) and in transit (TLS 1.3)
+
+## Data Flow Summary
+
+1. **Ingress**: Users upload FASTQ files via web UI → API Gateway → MinIO S3
+2. **Processing**: Jobs orchestrated via Kafka → Processing workers → Results stored in PostgreSQL/MongoDB
+3. **Egress**: Users download results via pre-signed S3 URLs (time-limited)
+4. **External**: Pipeline fetches reference data from NCBI/Ensembl on-demand (cached in MongoDB)
+
+---
+
+**Diagram Version**: 1.0
+**Last Updated**: 2025-10-12
+**Next Level**: [C4 Container Diagram](./c4-container.md)