agentic-flow 1.9.0 → 1.9.1

package/agentic-flow/examples/crispr-cas13-pipeline/docs/ARCHITECTURAL_DECISIONS.md

@@ -1,429 +0,0 @@
-# Architectural Decision Records (ADRs)
-
-## Overview
-
-This document records significant architectural decisions made for the CRISPR-Cas13 bioinformatics pipeline. Each decision includes context, alternatives considered, and the rationale for the chosen solution.
-
----
-
-## ADR-001: Microservices Architecture
-
-**Status**: Accepted
-**Date**: 2025-10-12
-**Decision Makers**: Architecture Agent, Technical Team
-
-### Context
-
-The CRISPR-Cas13 pipeline must process large volumes of genomic data with varying computational requirements (CPU-intensive alignment, memory-intensive differential expression, I/O-intensive data storage).
-
-### Decision
-
-Adopt a **microservices architecture** with independent services for each processing stage (alignment, off-target prediction, differential expression, immune response analysis).
-
-### Alternatives Considered
-
-1. **Monolithic Application**
-   - **Pros**: Simpler deployment, lower operational overhead
-   - **Cons**: Cannot scale components independently, single point of failure
-
-2. **Serverless Functions (AWS Lambda)**
-   - **Pros**: Auto-scaling, pay-per-execution
-   - **Cons**: 15-minute execution limit, cold starts, vendor lock-in
-
-3. **Microservices** ✅
-   - **Pros**: Independent scaling, fault isolation, technology flexibility
-   - **Cons**: Operational complexity, network overhead
-
-### Rationale
-
-- **Scalability**: Alignment service can scale to 20 replicas during peak load, while API gateway runs 3 replicas
-- **Fault Isolation**: Failure in off-target predictor doesn't affect differential expression
-- **Technology Flexibility**: Use R for statistical analysis, Python for ML, Rust for high-performance API
-
-### Consequences
-
-- **Positive**: 10x throughput improvement via horizontal scaling
-- **Negative**: Increased monitoring complexity (distributed tracing required)
-
----
-
-## ADR-002: Rust for API Gateway
-
-**Status**: Accepted
-**Date**: 2025-10-12
-
-### Context
-
-API gateway handles ~1000 requests/sec with strict latency requirements (P95 < 200ms).
-
-### Decision
-
-Implement API gateway in **Rust with Axum framework**.
-
-### Alternatives Considered
-
-1. **Node.js (Express/Fastify)**
-   - **Pros**: Large ecosystem, rapid development
-   - **Cons**: 5x slower than Rust, garbage collection pauses
-
-2. **Go (Gin/Echo)**
-   - **Pros**: Good performance, built-in concurrency
-   - **Cons**: Less type safety than Rust, no memory safety guarantees
-
-3. **Rust (Axum)** ✅
-   - **Pros**: ~10x faster than Node.js, memory safety, zero-cost abstractions
-   - **Cons**: Steeper learning curve, longer compile times
-
-### Rationale
-
-- **Performance**: Benchmarks show 5-10ms median latency vs. 50ms for Node.js
-- **Safety**: No null pointers, no data races, no buffer overflows
-- **Efficiency**: 100MB memory footprint vs. 500MB for Node.js
-
-### Consequences
-
-- **Positive**: Lower infrastructure costs (fewer pods needed)
-- **Negative**: Longer onboarding time for developers unfamiliar with Rust
-
----
-
-## ADR-003: Apache Kafka for Job Orchestration
-
-**Status**: Accepted
-**Date**: 2025-10-12
-
-### Context
-
-Need reliable message queue to decouple API from processing workers, handle job retries, and provide audit trail.
-
-### Decision
-
-Use **Apache Kafka** for job orchestration.
-
-### Alternatives Considered
-
-1. **RabbitMQ**
-   - **Pros**: Simpler setup, richer routing
-   - **Cons**: Lower throughput, no message replay
-
-2. **AWS SQS**
-   - **Pros**: Fully managed, auto-scaling
-   - **Cons**: Vendor lock-in, no message ordering
-
-3. **Apache Kafka** ✅
-   - **Pros**: High throughput (1M+ messages/sec), message replay, log-based architecture
-   - **Cons**: More complex setup and operations
-
-### Rationale
-
-- **Replay Capability**: Can re-process failed jobs from any offset
-- **Audit Trail**: All job events stored for 30 days
-- **Throughput**: Handles 10,000 jobs/day with <1s latency
-
-### Consequences
-
-- **Positive**: Zero job loss during system failures
-- **Negative**: Requires dedicated Kafka cluster (3+ brokers)
-
----
-
-## ADR-004: PostgreSQL + MongoDB Dual Database Strategy
-
-**Status**: Accepted
-**Date**: 2025-10-12
-
-### Context
-
-Data has both structured (experiments, samples, results) and semi-structured (genomic annotations, gene ontology) components.
-
-### Decision
-
-Use **PostgreSQL for relational data** and **MongoDB for document-based genomic data**.
-
-### Alternatives Considered
-
-1. **PostgreSQL Only (with JSONB)**
-   - **Pros**: Single database, ACID guarantees
-   - **Cons**: Poor performance for nested document queries
-
-2. **MongoDB Only**
-   - **Pros**: Flexible schema, horizontal scaling
-   - **Cons**: Weaker consistency, no foreign keys
-
-3. **PostgreSQL + MongoDB** ✅
-   - **Pros**: Best tool for each data type, optimized queries
-   - **Cons**: Two databases to maintain
-
-### Rationale
-
-- **PostgreSQL**: Perfect for experiments, samples, users (relational integrity critical)
-- **MongoDB**: Ideal for gene annotations (variable schema, nested documents)
-- **Performance**: PostgreSQL query time <50ms, MongoDB aggregation <100ms
-
-### Consequences
-
-- **Positive**: 3x faster genomic annotation queries
-- **Negative**: Data consistency requires distributed transactions (Saga pattern)
-
----
-
-## ADR-005: Kubernetes for Container Orchestration
-
-**Status**: Accepted
-**Date**: 2025-10-12
-
-### Context
-
-Need container orchestration platform with auto-scaling, self-healing, and multi-cloud support.
-
-### Decision
-
-Deploy on **Kubernetes (v1.28+)**.
-
-### Alternatives Considered
-
-1. **Docker Swarm**
-   - **Pros**: Simpler than Kubernetes
-   - **Cons**: Limited ecosystem, less robust auto-scaling
-
-2. **AWS ECS**
-   - **Pros**: Deep AWS integration, simpler than K8s
-   - **Cons**: Vendor lock-in, limited multi-cloud
-
-3. **Kubernetes** ✅
-   - **Pros**: Industry standard, rich ecosystem, multi-cloud
-   - **Cons**: Steep learning curve, operational complexity
-
-### Rationale
-
-- **Auto-Scaling**: HPA scales alignment service from 2 to 20 pods based on CPU and queue depth
-- **Self-Healing**: Crashed pods automatically restarted
-- **Multi-Cloud**: Can run on AWS, GCP, or on-premise with same manifests
-
-### Consequences
-
-- **Positive**: 99.9% uptime with rolling updates
-- **Negative**: Requires dedicated DevOps expertise
-
----
-
-## ADR-006: Redis for Caching & Rate Limiting
-
-**Status**: Accepted
-**Date**: 2025-10-12
-
-### Context
-
-Need sub-millisecond cache lookups for sessions, job statuses, and rate limiting.
-
-### Decision
-
-Use **Redis Cluster (6 nodes: 3 masters + 3 replicas)**.
-
-### Alternatives Considered
-
-1. **Memcached**
-   - **Pros**: Simpler, slightly faster
-   - **Cons**: No persistence, no complex data structures
-
-2. **In-Memory (HashMap)**
-   - **Pros**: Fastest (no network)
-   - **Cons**: Lost on pod restart, no sharing between pods
-
-3. **Redis** ✅
-   - **Pros**: Persistence, complex data types (sorted sets, hashes), pub/sub
-   - **Cons**: Slightly slower than Memcached
-
-### Rationale
-
-- **Performance**: GET operation <1ms
-- **Persistence**: AOF + RDB snapshots prevent data loss
-- **Rate Limiting**: Sliding window algorithm using sorted sets
-
-### Consequences
-
-- **Positive**: 50% reduction in database load
-- **Negative**: Redis cluster adds operational overhead
-
----
-
-## ADR-007: MinIO for Object Storage
-
-**Status**: Accepted
-**Date**: 2025-10-12
-
-### Context
-
-Store large sequencing files (FASTQ, BAM) totaling 50-100TB with S3-compatible API.
-
-### Decision
-
-Deploy **MinIO in distributed mode** (4 nodes × 4 drives, EC:4 erasure coding).
-
-### Alternatives Considered
-
-1. **AWS S3**
-   - **Pros**: Fully managed, 99.999999999% durability
-   - **Cons**: High egress costs ($0.09/GB), vendor lock-in
-
-2. **Ceph**
-   - **Pros**: Open source, proven at scale
-   - **Cons**: Complex setup, requires dedicated team
-
-3. **MinIO** ✅
-   - **Pros**: S3-compatible, easy deployment, lower cost
-   - **Cons**: Requires self-management
-
-### Rationale
-
-- **Cost**: $0.02/GB vs. $0.023/GB for S3 (no egress fees)
-- **Performance**: 10 Gbps throughput, <10ms latency
-- **Portability**: S3-compatible API, easy migration if needed
-
-### Consequences
-
-- **Positive**: 80% cost savings on storage
-- **Negative**: Must manage backups and disaster recovery
-
----
-
-## ADR-008: Prometheus + Grafana for Monitoring
-
-**Status**: Accepted
-**Date**: 2025-10-12
-
-### Context
-
-Need comprehensive monitoring of 50+ microservices, databases, and infrastructure.
-
-### Decision
-
-Use **Prometheus for metrics collection** and **Grafana for visualization**.
-
-### Alternatives Considered
-
-1. **Datadog**
-   - **Pros**: Fully managed, great UX
-   - **Cons**: Expensive ($15/host/month), vendor lock-in
-
-2. **ELK Stack (Elasticsearch + Kibana)**
-   - **Pros**: Powerful log analysis
-   - **Cons**: Resource-intensive, not optimized for metrics
-
-3. **Prometheus + Grafana** ✅
-   - **Pros**: Open source, Kubernetes-native, PromQL query language
-   - **Cons**: Requires self-hosting
-
-### Rationale
-
-- **Integration**: Native Kubernetes support, service discovery
-- **Query Language**: PromQL enables complex aggregations
-- **Cost**: Free (self-hosted) vs. $10,000/month for Datadog
-
-### Consequences
-
-- **Positive**: 100% cost savings vs. SaaS monitoring
-- **Negative**: Must manage Prometheus/Grafana infrastructure
-
----
-
-## ADR-009: JWT with RS256 for Authentication
-
-**Status**: Accepted
-**Date**: 2025-10-12
-
-### Context
-
-Need stateless authentication for REST API with role-based access control.
-
-### Decision
-
-Use **JWT tokens signed with RS256** (RSA public/private key).
-
-### Alternatives Considered
-
-1. **Session Cookies (Server-Side)**
-   - **Pros**: Simpler revocation
-   - **Cons**: Requires sticky sessions or shared session store
-
-2. **JWT with HS256 (HMAC)**
-   - **Pros**: Simpler (symmetric key)
-   - **Cons**: All services need secret key (security risk)
-
-3. **JWT with RS256** ✅
-   - **Pros**: Public key distribution via JWKS, stateless validation
-   - **Cons**: Slightly slower than HS256
-
-### Rationale
-
-- **Stateless**: No database lookup on every request
-- **Security**: Private key only on auth service, public key distributed
-- **Scalability**: API gateway validates tokens without auth service
-
-### Consequences
-
-- **Positive**: <1ms token validation overhead
-- **Negative**: Token revocation requires blacklist (Redis)
-
----
-
-## ADR-010: React + TypeScript for Frontend
-
-**Status**: Accepted
-**Date**: 2025-10-12
-
-### Context
-
-Need modern, interactive UI for experiment management and result visualization.
-
-### Decision
-
-Build frontend with **React 18 + TypeScript + Vite**.
-
-### Alternatives Considered
-
-1. **Vue.js**
-   - **Pros**: Simpler learning curve, better docs
-   - **Cons**: Smaller ecosystem, fewer enterprise adoptions
-
-2. **Svelte**
-   - **Pros**: Smaller bundle size, faster runtime
-   - **Cons**: Smaller ecosystem, fewer third-party libraries
-
-3. **React + TypeScript** ✅
-   - **Pros**: Largest ecosystem, strong typing, component reusability
-   - **Cons**: Larger bundle size (mitigated by code splitting)
-
-### Rationale
-
-- **Ecosystem**: Best visualization libraries (D3.js, Plotly, IGV.js)
-- **Type Safety**: TypeScript prevents runtime errors
-- **Developer Experience**: Vite provides <1s HMR
-
-### Consequences
-
-- **Positive**: Faster development with component libraries (Shadcn/ui)
-- **Negative**: 500KB bundle size (gzipped: 150KB)
-
----
-
-## Summary Table
-
-| ADR | Decision | Status | Impact |
-|-----|----------|--------|--------|
-| ADR-001 | Microservices Architecture | Accepted | 10x throughput |
-| ADR-002 | Rust for API Gateway | Accepted | 5-10x faster |
-| ADR-003 | Apache Kafka | Accepted | Zero job loss |
-| ADR-004 | PostgreSQL + MongoDB | Accepted | 3x faster queries |
-| ADR-005 | Kubernetes | Accepted | 99.9% uptime |
-| ADR-006 | Redis Cluster | Accepted | 50% DB load reduction |
-| ADR-007 | MinIO | Accepted | 80% cost savings |
-| ADR-008 | Prometheus + Grafana | Accepted | 100% cost savings |
-| ADR-009 | JWT with RS256 | Accepted | <1ms validation |
-| ADR-010 | React + TypeScript | Accepted | Faster development |
-
----
-
-**Document Version**: 1.0
-**Last Updated**: 2025-10-12
-**See Also**: [Architecture Overview](./ARCHITECTURE.md)