agentic-flow 1.9.0 → 1.9.2

package/agentic-flow/examples/crispr-cas13-pipeline/docs/architecture-diagrams/c4-component.md

@@ -1,240 +0,0 @@
-# C4 Component Diagram - API Gateway & Processing Services
-
-## API Gateway Components
-
-This diagram shows the internal structure of the API Gateway container.
-
-```mermaid
-C4Component
-    title Component Diagram - API Gateway (Axum/Rust)
-
-    Container_Boundary(api_gateway, "API Gateway") {
-        Component(router, "Router", "Axum Router", "Routes HTTP requests to handlers")
-        Component(auth_middleware, "Auth Middleware", "Tower Layer", "JWT validation, permission checks")
-        Component(rate_limiter, "Rate Limiter", "Tower Layer", "Redis-backed sliding window")
-        Component(logger, "Request Logger", "Tower Layer", "Structured logging with tracing")
-
-        Component(auth_handler, "Auth Handler", "Axum Handler", "Login, logout, token refresh")
-        Component(experiment_handler, "Experiment Handler", "Axum Handler", "CRUD for experiments")
-        Component(sample_handler, "Sample Handler", "Axum Handler", "Upload FASTQ files")
-        Component(job_handler, "Job Handler", "Axum Handler", "Submit and monitor jobs")
-        Component(result_handler, "Result Handler", "Axum Handler", "Query analysis results")
-
-        Component(db_service, "Database Service", "SQLx", "PostgreSQL connection pool")
-        Component(cache_service, "Cache Service", "redis-rs", "Redis client")
-        Component(kafka_service, "Kafka Service", "rdkafka", "Message producer")
-        Component(storage_service, "Storage Service", "rusoto_s3", "MinIO S3 client")
-    }
-
-    ContainerDb(postgres, "PostgreSQL", "Database")
-    ContainerDb(redis, "Redis", "Cache")
-    ContainerQueue(kafka, "Kafka", "Message Queue")
-    ContainerDb(minio, "MinIO", "Object Storage")
-
-    Rel(router, auth_middleware, "Applies", "Layer")
-    Rel(router, rate_limiter, "Applies", "Layer")
-    Rel(router, logger, "Applies", "Layer")
-
-    Rel(auth_middleware, redis, "Verify session", "TCP")
-
-    Rel(router, auth_handler, "Routes /auth/*")
-    Rel(router, experiment_handler, "Routes /experiments/*")
-    Rel(router, sample_handler, "Routes /samples/*")
-    Rel(router, job_handler, "Routes /jobs/*")
-    Rel(router, result_handler, "Routes /results/*")
-
-    Rel(auth_handler, db_service, "Uses")
-    Rel(experiment_handler, db_service, "Uses")
-    Rel(sample_handler, storage_service, "Uses")
-    Rel(job_handler, kafka_service, "Uses")
-    Rel(result_handler, db_service, "Uses")
-
-    Rel(db_service, postgres, "Queries", "TCP/5432")
-    Rel(cache_service, redis, "Get/Set", "TCP/6379")
-    Rel(kafka_service, kafka, "Publish", "TCP/9092")
-    Rel(storage_service, minio, "Upload/Download", "HTTPS/9000")
-
-    UpdateLayoutConfig($c4ShapeInRow="4", $c4BoundaryInRow="1")
-```
-
-## API Gateway Component Details
-
-### 1. Middleware Stack
-
-**Execution Order** (outermost to innermost):
-1. **Request Logger**: Logs all incoming requests (method, path, headers)
-2. **Rate Limiter**: Checks Redis for rate limit, returns 429 if exceeded
-3. **Auth Middleware**: Validates JWT token, extracts user claims
-4. **Router**: Routes request to appropriate handler
-
-**Tower Middleware Example**:
-```rust
-let app = Router::new()
-    .nest("/api/v1", api_routes)
-    .layer(
-        ServiceBuilder::new()
-            .layer(TraceLayer::new_for_http())  // Request logger
-            .layer(middleware::from_fn(rate_limit_middleware))
-            .layer(middleware::from_fn(auth_middleware))
-    );
-```
-
-### 2. Route Handlers
-
-**Auth Handler** (`/api/v1/auth/*`)
-- `POST /login`: Username/password → JWT token
-- `POST /logout`: Invalidate session
-- `POST /refresh`: Refresh token → New JWT
-- `GET /verify`: Validate current token
-
-**Experiment Handler** (`/api/v1/experiments/*`)
-- `GET /`: List experiments (filtered by user)
-- `POST /`: Create new experiment
-- `GET /:id`: Get experiment details
-- `PUT /:id`: Update experiment
-- `DELETE /:id`: Delete experiment
-
-**Sample Handler** (`/api/v1/samples/*`)
-- `POST /upload`: Upload FASTQ files to MinIO
-- `GET /:id`: Get sample metadata
-- `DELETE /:id`: Delete sample
-
-**Job Handler** (`/api/v1/jobs/*`)
-- `POST /`: Submit analysis job to Kafka
-- `GET /:id`: Get job status
-- `GET /:id/logs`: Stream job logs
-- `DELETE /:id`: Cancel running job
-
-**Result Handler** (`/api/v1/results/*`)
-- `GET /off-targets/:job_id`: Query off-target predictions
-- `GET /diff-expr/:job_id`: Query differential expression results
-- `GET /immune/:job_id`: Query immune response signatures
-- `GET /export/:job_id`: Generate CSV/JSON export
-
-### 3. Service Layer
-
-**Database Service**
-- **Connection Pool**: 100 max connections (SQLx)
-- **Query Builder**: Type-safe queries with `sqlx::query!` macro
-- **Transactions**: Automatic rollback on error
-- **Migrations**: Versioned schema migrations
-
-**Cache Service**
-- **Client**: redis-rs with connection pooling
-- **Patterns**:
-  - Session storage (`session:{token}`)
-  - Rate limiting (`rate_limit:{user}:{minute}`)
-  - Job status (`job:{id}`)
-- **TTL**: Configurable per key (default: 5 minutes)
-
-**Kafka Service**
-- **Producer**: rdkafka FutureProducer (async)
-- **Serialization**: JSON with `serde_json`
-- **Topics**: `jobs.alignment`, `jobs.off_target`, `jobs.diff_expr`, `jobs.immune`
-- **Error Handling**: Retry 3 times with exponential backoff
-
-**Storage Service**
-- **Client**: rusoto_s3 (AWS SDK for Rust)
-- **Buckets**: `crispr-cas13` (main), `crispr-cas13-backups`
-- **Pre-signed URLs**: 1-hour TTL for downloads
-- **Multipart Upload**: For files >5GB
-
----
-
-## Processing Service Components (Alignment Service)
-
-```mermaid
-C4Component
-    title Component Diagram - Alignment Service (Python/Bowtie2)
-
-    Container_Boundary(alignment_service, "Alignment Service") {
-        Component(kafka_consumer, "Kafka Consumer", "kafka-python", "Consumes alignment jobs")
-        Component(file_downloader, "File Downloader", "boto3", "Downloads FASTQ from MinIO")
-        Component(bowtie2_wrapper, "Bowtie2 Wrapper", "subprocess", "Runs Bowtie2 alignment")
-        Component(samtools_wrapper, "Samtools Wrapper", "subprocess", "BAM conversion & indexing")
-        Component(qc_module, "QC Module", "FastQC/MultiQC", "Quality control reports")
-        Component(file_uploader, "File Uploader", "boto3", "Uploads BAM to MinIO")
-        Component(db_updater, "DB Updater", "psycopg2", "Updates job status in PostgreSQL")
-        Component(metrics_collector, "Metrics Collector", "prometheus_client", "Exposes metrics")
-    }
-
-    ContainerQueue(kafka, "Kafka", "Message Queue")
-    ContainerDb(minio, "MinIO", "Object Storage")
-    ContainerDb(postgres, "PostgreSQL", "Database")
-
-    Rel(kafka, kafka_consumer, "Delivers jobs", "TCP/9092")
-    Rel(kafka_consumer, file_downloader, "Triggers")
-    Rel(file_downloader, minio, "Download FASTQ", "HTTPS/9000")
-
-    Rel(kafka_consumer, bowtie2_wrapper, "Executes")
-    Rel(bowtie2_wrapper, samtools_wrapper, "Pipes SAM")
-    Rel(samtools_wrapper, qc_module, "Generates stats")
-
-    Rel(qc_module, file_uploader, "Upload BAM + reports")
-    Rel(file_uploader, minio, "Upload", "HTTPS/9000")
-
-    Rel(kafka_consumer, db_updater, "Update status")
-    Rel(db_updater, postgres, "INSERT/UPDATE", "TCP/5432")
-
-    Rel(kafka_consumer, metrics_collector, "Record metrics")
-
-    UpdateLayoutConfig($c4ShapeInRow="4", $c4BoundaryInRow="1")
-```
-
-## Alignment Service Workflow
-
-1. **Consume Job**: Kafka consumer receives alignment job message
-2. **Download FASTQ**: Download R1/R2 FASTQ files from MinIO to `/tmp`
-3. **Run Bowtie2**: Execute Bowtie2 alignment (8 threads)
-   ```bash
-   bowtie2 -x ref_genome -1 R1.fastq.gz -2 R2.fastq.gz -p 8 --very-sensitive
-   ```
-4. **Convert to BAM**: Pipe SAM output to samtools
-   ```bash
-   samtools view -bS - | samtools sort -o sorted.bam
-   samtools index sorted.bam
-   ```
-5. **Quality Control**: Run FastQC on BAM file
-6. **Upload Results**: Upload BAM, BAM index, QC report to MinIO
-7. **Update Database**: Mark job as completed in PostgreSQL
-8. **Publish Event**: Send completion message to Kafka (`jobs.completed`)
-9. **Cleanup**: Delete temporary files from `/tmp`
-
-## Error Handling Strategy
-
-**Transient Errors** (network failures, timeouts):
-- **Retry**: 3 attempts with exponential backoff (1s, 2s, 4s)
-- **Dead Letter Queue**: After 3 failures, move to `jobs.failed` topic
-
-**Permanent Errors** (invalid input, out of memory):
-- **Immediate Failure**: Mark job as failed in database
-- **Notification**: Send alert to user via email/WebSocket
-- **Logging**: Store detailed error message in PostgreSQL
-
-**Resource Exhaustion**:
-- **Graceful Degradation**: If disk space <10%, pause processing
-- **Auto-Recovery**: Resume when space is freed
-
----
-
-## Component Communication Patterns
-
-### Synchronous (Request-Response)
-- **API Gateway → PostgreSQL**: Direct database queries (SELECT, INSERT, UPDATE)
-- **API Gateway → Redis**: Cache lookups (GET, SET)
-- **API Gateway → MinIO**: File operations (PUT, GET)
-
-### Asynchronous (Event-Driven)
-- **API Gateway → Kafka**: Job submission (fire-and-forget)
-- **Kafka → Workers**: Job consumption (at-least-once delivery)
-- **Workers → Kafka**: Status updates (completion/failure events)
-
-### Streaming
-- **WebSocket Server → Clients**: Real-time job status updates
-- **Prometheus → Grafana**: Continuous metrics streaming
-
----
-
-**Diagram Version**: 1.0
-**Last Updated**: 2025-10-12
-**See Also**: [Sequence Diagrams](./sequence-diagrams.md)

package/agentic-flow/examples/crispr-cas13-pipeline/docs/architecture-diagrams/c4-container.md

@@ -1,223 +0,0 @@
-# C4 Container Diagram - CRISPR-Cas13 Pipeline
-
-## Container Architecture
-
-This diagram shows the high-level technology choices and how containers communicate.
-
-```mermaid
-C4Container
-    title Container Diagram - CRISPR-Cas13 Pipeline
-
-    Person(user, "User", "Researcher or Bioinformatician")
-
-    Container_Boundary(frontend, "Client Applications") {
-        Container(web_app, "Web Application", "React/TypeScript", "Interactive UI for experiment management")
-        Container(cli_tool, "CLI Tool", "Python/Rust", "Command-line interface for power users")
-    }
-
-    Container_Boundary(api_layer, "API Layer") {
-        Container(nginx, "Nginx Ingress", "Reverse Proxy", "Load balancing, TLS termination")
-        Container(api_gateway, "API Gateway", "Axum (Rust)", "RESTful API, authentication, rate limiting")
-        Container(websocket_server, "WebSocket Server", "Axum (Rust)", "Real-time job status updates")
-    }
-
-    Container_Boundary(processing_layer, "Processing Layer") {
-        Container(alignment_service, "Alignment Service", "Python/Bowtie2", "Read alignment to genome")
-        Container(off_target_service, "Off-Target Predictor", "Python/PyTorch", "ML-based off-target prediction")
-        Container(diff_expr_service, "Differential Expression", "R/DESeq2", "Gene expression analysis")
-        Container(immune_service, "Immune Response Analyzer", "Python/R", "Signature enrichment analysis")
-    }
-
-    Container_Boundary(data_layer, "Data Layer") {
-        ContainerDb(postgres, "PostgreSQL", "Relational DB", "Experiments, samples, results")
-        ContainerDb(mongodb, "MongoDB", "Document DB", "Genomic annotations, gene ontology")
-        ContainerDb(redis, "Redis", "Cache", "Sessions, job status, rate limiting")
-        ContainerDb(minio, "MinIO", "Object Storage", "FASTQ, BAM, VCF files")
-    }
-
-    Container_Boundary(messaging, "Message Queue") {
-        ContainerQueue(kafka, "Apache Kafka", "Message Broker", "Job orchestration, event streaming")
-    }
-
-    Container_Boundary(monitoring, "Monitoring") {
-        Container(prometheus, "Prometheus", "Metrics", "Time-series metrics collection")
-        Container(grafana, "Grafana", "Dashboards", "Visualization and alerting")
-        Container(jaeger, "Jaeger", "Tracing", "Distributed request tracing")
-    }
-
-    Rel(user, web_app, "Uses", "HTTPS")
-    Rel(user, cli_tool, "Uses", "HTTPS")
-
-    Rel(web_app, nginx, "API requests", "HTTPS/443")
-    Rel(cli_tool, nginx, "API requests", "HTTPS/443")
-    Rel(nginx, api_gateway, "Routes", "HTTP/2")
-    Rel(nginx, websocket_server, "WebSocket", "WSS")
-
-    Rel(api_gateway, postgres, "Reads/Writes", "TCP/5432")
-    Rel(api_gateway, redis, "Cache", "TCP/6379")
-    Rel(api_gateway, kafka, "Publishes jobs", "TCP/9092")
-    Rel(api_gateway, minio, "Upload/Download", "HTTPS/9000")
-
-    Rel(websocket_server, kafka, "Subscribes to updates", "TCP/9092")
-    Rel(websocket_server, redis, "Job status", "TCP/6379")
-
-    Rel(kafka, alignment_service, "Consumes", "TCP/9092")
-    Rel(kafka, off_target_service, "Consumes", "TCP/9092")
-    Rel(kafka, diff_expr_service, "Consumes", "TCP/9092")
-    Rel(kafka, immune_service, "Consumes", "TCP/9092")
-
-    Rel(alignment_service, minio, "Read FASTQ, Write BAM", "HTTPS/9000")
-    Rel(alignment_service, postgres, "Update job status", "TCP/5432")
-    Rel(alignment_service, kafka, "Publish completion", "TCP/9092")
-
-    Rel(off_target_service, minio, "Read BAM", "HTTPS/9000")
-    Rel(off_target_service, postgres, "Write off-targets", "TCP/5432")
-    Rel(off_target_service, mongodb, "Query genome", "TCP/27017")
-
-    Rel(diff_expr_service, minio, "Read BAM", "HTTPS/9000")
-    Rel(diff_expr_service, postgres, "Write results", "TCP/5432")
-
-    Rel(immune_service, postgres, "Read diff expr", "TCP/5432")
-    Rel(immune_service, mongodb, "Query signatures", "TCP/27017")
-
-    Rel(api_gateway, prometheus, "Metrics", "HTTP/9090")
-    Rel(alignment_service, prometheus, "Metrics", "HTTP/9090")
-    Rel(prometheus, grafana, "Queries", "HTTP/3000")
-    Rel(api_gateway, jaeger, "Traces", "UDP/6831")
-
-    UpdateLayoutConfig($c4ShapeInRow="3", $c4BoundaryInRow="2")
-```
-
-## Container Communication Matrix
-
-| Source Container | Target Container | Protocol | Port | Purpose | Security |
-|------------------|------------------|----------|------|---------|----------|
-| Web App → Nginx | API Gateway | HTTPS | 443 | REST API calls | TLS 1.3, JWT |
-| API Gateway → PostgreSQL | Database | TCP | 5432 | CRUD operations | TLS, user/pass |
-| API Gateway → Redis | Cache | TCP | 6379 | Session storage | TLS, password |
-| API Gateway → Kafka | Message Queue | TCP | 9092 | Job submission | SASL/SSL |
-| API Gateway → MinIO | Object Storage | HTTPS | 9000 | File upload/download | IAM keys |
-| Kafka → Workers | Processing | TCP | 9092 | Job distribution | SASL/SSL |
-| Workers → MinIO | Object Storage | HTTPS | 9000 | Read/Write files | IAM keys |
-| Workers → PostgreSQL | Database | TCP | 5432 | Update results | TLS, user/pass |
-| Workers → MongoDB | Document DB | TCP | 27017 | Query annotations | TLS, user/pass |
-| All → Prometheus | Monitoring | HTTP | 9090 | Metrics scraping | Service account |
-
-## Container Technologies
-
-### Frontend Layer
-
-**Web Application**
-- **Framework**: React 18 + TypeScript 5.3
-- **Build Tool**: Vite 5.0
-- **State Management**: Zustand
-- **Deployment**: Nginx static hosting
-- **CDN**: CloudFlare
-
-**CLI Tool**
-- **Language**: Python 3.11 (alternative: Rust)
-- **HTTP Client**: `requests` / `httpx`
-- **Distribution**: PyPI package
-
-### API Layer
-
-**API Gateway**
-- **Framework**: Axum 0.7 (Rust)
-- **Async Runtime**: Tokio 1.35
-- **Middleware**: Tower
-- **Serialization**: Serde
-- **Database Client**: SQLx (PostgreSQL)
-- **Cache Client**: redis-rs
-- **Message Queue**: rdkafka
-- **Deployment**: Docker container on Kubernetes
-
-**WebSocket Server**
-- **Framework**: Axum WebSocket support
-- **Broadcasting**: Tokio broadcast channels
-- **Deployment**: Shared pods with API Gateway
-
-### Processing Layer
-
-**Alignment Service**
-- **Language**: Python 3.11
-- **Bioinformatics**: Bioconda (Bowtie2, Samtools)
-- **Message Queue**: kafka-python
-- **Deployment**: Docker container with Bioconda
-
-**Off-Target Service**
-- **Language**: Python 3.11
-- **ML Framework**: PyTorch 2.1
-- **Genomics**: Biopython
-- **Deployment**: Docker container (GPU-enabled optional)
-
-**Differential Expression Service**
-- **Language**: R 4.3
-- **Framework**: Bioconductor 3.18
-- **Packages**: DESeq2, edgeR, ggplot2
-- **Deployment**: Docker container with R/Bioconductor
-
-**Immune Response Service**
-- **Language**: Python 3.11 + R 4.3 (hybrid)
-- **Packages**: GSEA, MSigDB, ImmuneSigDB
-- **Deployment**: Docker container
-
-### Data Layer
-
-**PostgreSQL**
-- **Version**: 16
-- **Configuration**: Primary + 3 read replicas
-- **Connection Pooling**: PgBouncer
-- **Backup**: Automated daily snapshots
-
-**MongoDB**
-- **Version**: 7.0
-- **Sharding**: 3 shards × 3 replicas
-- **Indexes**: Compound indexes on chromosome + position
-
-**Redis**
-- **Version**: 7.2
-- **Mode**: Cluster (3 masters + 3 replicas)
-- **Persistence**: AOF + RDB snapshots
-
-**MinIO**
-- **Mode**: Distributed (4 nodes × 4 drives)
-- **Erasure Coding**: EC:4 (4 data + 4 parity)
-- **Replication**: Cross-region replication
-
-### Message Queue
-
-**Apache Kafka**
-- **Version**: 3.6
-- **Brokers**: 3 brokers (HA)
-- **Replication**: Factor 3
-- **Partitions**: 10 per topic
-
-### Monitoring
-
-**Prometheus**
-- **Retention**: 30 days
-- **Scrape Interval**: 15 seconds
-- **Storage**: 500GB SSD
-
-**Grafana**
-- **Dashboards**: 5 pre-built dashboards
-- **Alerting**: Prometheus Alertmanager
-
-**Jaeger**
-- **Backend**: Elasticsearch
-- **Retention**: 7 days
-- **Sampling**: 10% of requests
-
-## Deployment Model
-
-All containers run on **Kubernetes 1.28** with:
-- **Namespace Isolation**: `crispr-production`, `crispr-staging`, `crispr-dev`
-- **Resource Quotas**: Per-namespace CPU/memory limits
-- **Network Policies**: Restrict inter-service communication
-- **Ingress Controller**: Nginx Ingress with Let's Encrypt
-
----
-
-**Diagram Version**: 1.0
-**Last Updated**: 2025-10-12
-**Next Level**: [C4 Component Diagram](./c4-component.md)

package/agentic-flow/examples/crispr-cas13-pipeline/docs/architecture-diagrams/c4-context.md

@@ -1,86 +0,0 @@
-# C4 Context Diagram - CRISPR-Cas13 Pipeline
-
-## System Context
-
-This diagram shows the CRISPR-Cas13 Analysis Pipeline in the context of external users and systems.
-
-```mermaid
-C4Context
-    title System Context Diagram - CRISPR-Cas13 Analysis Pipeline
-
-    Person(researcher, "Research Scientist", "Designs experiments, analyzes results")
-    Person(bioinformatician, "Bioinformatician", "Configures pipelines, validates results")
-    Person(admin, "System Administrator", "Monitors health, manages resources")
-
-    System(crispr_pipeline, "CRISPR-Cas13 Pipeline", "Analyzes off-target effects and immune responses from CRISPR-Cas13 experiments")
-
-    System_Ext(ncbi, "NCBI Database", "Reference genomes and annotations")
-    System_Ext(ensembl, "Ensembl Genome Browser", "Gene annotations and orthologs")
-    System_Ext(immunesigdb, "ImmuneSigDB", "Immune response gene signatures")
-    System_Ext(hpc, "HPC Cluster", "High-performance computing resources")
-
-    Rel(researcher, crispr_pipeline, "Submits samples, views results", "HTTPS")
-    Rel(bioinformatician, crispr_pipeline, "Configures pipelines, exports data", "HTTPS/CLI")
-    Rel(admin, crispr_pipeline, "Monitors system, manages users", "HTTPS/SSH")
-
-    Rel(crispr_pipeline, ncbi, "Downloads reference genomes", "FTP/HTTPS")
-    Rel(crispr_pipeline, ensembl, "Queries gene annotations", "REST API")
-    Rel(crispr_pipeline, immunesigdb, "Retrieves signature data", "REST API")
-    Rel(crispr_pipeline, hpc, "Offloads compute-intensive jobs", "SSH/Slurm")
-
-    UpdateLayoutConfig($c4ShapeInRow="3", $c4BoundaryInRow="1")
-```
-
-## External Systems Integration
-
-### 1. NCBI Database
-- **Purpose**: Download reference genomes (e.g., Macaca mulatta rheMac10)
-- **Protocol**: FTP, HTTPS
-- **Frequency**: Monthly (genome updates)
-- **Authentication**: Public access, API key for higher rate limits
-
-### 2. Ensembl Genome Browser
-- **Purpose**: Gene annotations, transcript variants, orthologs
-- **Protocol**: REST API (`https://rest.ensembl.org/`)
-- **Frequency**: Weekly updates
-- **Rate Limit**: 15 requests/sec
-
-### 3. ImmuneSigDB
-- **Purpose**: Curated immune response gene signatures
-- **Protocol**: REST API, bulk download
-- **Frequency**: Quarterly updates
-- **Data Format**: GMT, JSON
-
-### 4. HPC Cluster (Optional)
-- **Purpose**: Offload large-scale alignment jobs
-- **Protocol**: SSH, Slurm job scheduler
-- **Integration**: Job submission via `sbatch`, result retrieval via `scp`
-- **Use Case**: Experiments with >50 samples
-
-## User Roles & Responsibilities
-
-| Role | Responsibilities | Access Level |
-|------|-----------------|--------------|
-| **Research Scientist** | - Design experiments<br/>- Submit samples<br/>- Interpret results | Read/Write own experiments |
-| **Bioinformatician** | - Configure analysis pipelines<br/>- Validate results<br/>- Export raw data | Read all, Write pipelines |
-| **System Administrator** | - Monitor system health<br/>- Manage users and permissions<br/>- Perform backups | Full system access |
-
-## Security Boundaries
-
-- **External Firewall**: All external traffic goes through firewall (HTTPS only)
-- **API Gateway**: Authentication layer (OAuth2/JWT)
-- **Internal Network**: Service-to-service communication uses mTLS
-- **Data Encryption**: All data encrypted at rest (AES-256) and in transit (TLS 1.3)
-
-## Data Flow Summary
-
-1. **Ingress**: Users upload FASTQ files via web UI → API Gateway → MinIO S3
-2. **Processing**: Jobs orchestrated via Kafka → Processing workers → Results stored in PostgreSQL/MongoDB
-3. **Egress**: Users download results via pre-signed S3 URLs (time-limited)
-4. **External**: Pipeline fetches reference data from NCBI/Ensembl on-demand (cached in MongoDB)
-
----
-
-**Diagram Version**: 1.0
-**Last Updated**: 2025-10-12
-**Next Level**: [C4 Container Diagram](./c4-container.md)