agentic-flow 1.9.0 → 1.9.1

package/agentic-flow/examples/crispr-cas13-pipeline/docs/SPECIFICATION.md

@@ -1,719 +0,0 @@
-# CRISPR-Cas13 Bioinformatics Pipeline - System Requirements Specification
-
-**Version:** 1.0.0
-**Date:** 2025-10-12
-**Phase:** SPARC Specification
-**Status:** Draft
-
----
-
-## 1. Introduction
-
-### 1.1 Purpose
-This document specifies the requirements for a bioinformatics pipeline system designed to process CRISPR-Cas13 RNA-targeting experiments in primate models, with focus on innate immune response analysis, off-target prediction, and experimental provenance tracking.
-
-### 1.2 Scope
-The system encompasses:
-- **Data Ingestion**: FASTQ sequencing data, experimental metadata, sample annotations
-- **Processing Pipeline**: Quality control, alignment, target identification, off-target prediction
-- **Analysis**: Immune pathway gene expression analysis, statistical validation
-- **Storage**: PostgreSQL for relational data, MongoDB for genomic data
-- **API**: RESTful endpoints with real-time monitoring via WebSocket
-- **Provenance**: Complete experimental lineage tracking per GA4GH standards
-
-### 1.3 Definitions
-- **CRISPR-Cas13**: RNA-targeting CRISPR system for transcript knockdown
-- **Off-target**: Unintended RNA cleavage sites with sequence similarity to guide RNA
-- **Innate Immune Response**: Type I/III interferon pathway activation
-- **Provenance**: Complete audit trail of data processing and analysis steps
-- **Guide RNA (gRNA)**: 22-30nt RNA sequence directing Cas13 to target
-- **FASTQ**: Standard format for raw sequencing reads with quality scores
-- **BAM/SAM**: Binary/text alignment format (GA4GH standard)
-
-### 1.4 Bioinformatics Standards Compliance
-- **GA4GH**: Global Alliance for Genomics and Health data standards
-- **NCBI**: GenBank/RefSeq sequence formats
-- **SAM/BAM**: Sequence Alignment/Map format specification v1.6
-- **VCF**: Variant Call Format v4.3 for variant annotations
-- **MIAME**: Minimum Information About a Microarray Experiment (adapted for RNA-seq)
-
----
-
-## 2. Functional Requirements
-
-### 2.1 Data Ingestion (FR-2.1)
-
-#### FR-2.1.1: FASTQ Data Upload
-**Priority:** Critical
-**Description:** System shall accept paired-end FASTQ files (R1/R2) with Phred33 quality encoding
-
-**Acceptance Criteria:**
-- Support gzip compressed FASTQ (.fastq.gz)
-- Validate FASTQ format per Illumina/ONT specifications
-- Maximum file size: 50GB per file
-- Checksum validation (MD5/SHA256)
-- Automatic metadata extraction from FASTQ headers
-
-**Validation Rules:**
-```yaml
-fastq_validation:
-  - header_format: "@instrument:run:flowcell:lane:tile:x:y read:filter:control:index"
-  - quality_encoding: "Phred33 (ASCII 33-126)"
-  - min_read_length: 50
-  - max_read_length: 300
-  - allowed_bases: "ACGTN"
-```
-
-#### FR-2.1.2: Experimental Metadata Ingestion
-**Priority:** Critical
-**Description:** Accept structured experimental metadata in JSON/YAML format
-
-**Required Fields:**
-```yaml
-experiment:
-  experiment_id: string (UUID)
-  title: string
-  description: text
-  species: enum [rhesus_macaque, cynomolgus_macaque, human_cell_line]
-  tissue_type: enum [liver, kidney, brain, blood, cultured_cells]
-  cas13_variant: enum [Cas13a, Cas13b, Cas13c, Cas13d]
-  guide_rna:
-    sequence: string (22-30nt)
-    target_gene: string (HGNC symbol)
-    target_transcript_id: string (Ensembl/RefSeq)
-  delivery_method: enum [AAV, LNP, electroporation]
-  timepoint_hours: integer
-  replicate_number: integer
-  experimenter: string
-  institution: string
-  date_performed: ISO8601
-```
-
-#### FR-2.1.3: Reference Genome Management
-**Priority:** High
-**Description:** Support multiple reference genomes with automatic indexing
-
-**Supported References:**
-- *Macaca mulatta* (rheMac10)
-- *Macaca fascicularis* (macFas5)
-- *Homo sapiens* (GRCh38/hg38)
-- Custom transcriptome assemblies
-
-### 2.2 Processing Pipeline (FR-2.2)
-
-#### FR-2.2.1: Quality Control
-**Priority:** Critical
-**Description:** Perform automated QC on raw sequencing data
-
-**QC Metrics:**
-- Per-base sequence quality (FastQC)
-- Adapter contamination detection
-- Overrepresented sequences
-- GC content distribution
-- Duplication rate
-- Read length distribution
-
-**Acceptance Criteria:**
-- Generate QC report in HTML/JSON format
-- Flag samples with mean Phred < 28
-- Automatic adapter trimming if >5% contamination
-
-#### FR-2.2.2: Read Alignment
-**Priority:** Critical
-**Description:** Align reads to reference transcriptome using STAR or HISAT2
-
-**Alignment Parameters:**
-```yaml
-alignment:
-  aligner: STAR
-  max_mismatches: 2
-  min_intron_length: 20
-  max_intron_length: 1000000
-  output_format: BAM (coordinate-sorted)
-  secondary_alignments: true
-  supplementary_alignments: true
-```
-
-**Acceptance Criteria:**
-- Generate BAM/BAI index files
-- Calculate alignment statistics (mapped %, multimappers)
-- Store alignment metrics in database
-
-#### FR-2.2.3: Target Site Identification
-**Priority:** Critical
-**Description:** Identify Cas13 target sites and quantify knockdown efficiency
-
-**Analysis:**
-- Map guide RNA to transcriptome with exact/mismatch search
-- Quantify transcript abundance (TPM/RPKM)
-- Calculate knockdown ratio vs control samples
-- Statistical significance testing (DESeq2/edgeR)
-
-#### FR-2.2.4: Off-Target Prediction
-**Priority:** High
-**Description:** Predict potential off-target sites genome-wide
-
-**Algorithm Requirements:**
-- Seed-based search (12-15nt seed region)
-- Mismatch tolerance: 0-4 mismatches
-- Position-weighted scoring (PAM-proximal > distal)
-- Sequence context analysis (secondary structure)
-
-**Confidence Scoring:**
-```python
-off_target_score = (
-    0.4 * seed_match_score +
-    0.3 * full_sequence_similarity +
-    0.2 * accessibility_score +
-    0.1 * expression_level
-)
-```
-
-**Output:**
-- Ranked list of off-target sites (top 1000)
-- Confidence score: 0.0-1.0
-- Genomic coordinates (BED format)
-- Overlapping gene annotations
-
-#### FR-2.2.5: Immune Response Analysis
-**Priority:** High
-**Description:** Quantify innate immune pathway activation
-
-**Target Pathways:**
-- Type I interferon signaling (IRF3/7, STAT1/2, ISGs)
-- Type III interferon signaling (IL28/29, IFNλR)
-- RIG-I/MDA5 pathway (DDX58, IFIH1, MAVS)
-- OAS/RNase L pathway (OAS1/2/3, RNASEL)
-- PKR pathway (EIF2AK2)
-
-**Analysis:**
-- Gene set enrichment analysis (GSEA)
-- Differential expression testing (adjusted p < 0.05)
-- Fold-change calculation vs mock-treated controls
-- Heatmap generation for immune gene panels
-
-### 2.3 Job Management (FR-2.3)
-
-#### FR-2.3.1: Job Submission
-**Priority:** Critical
-**Description:** Submit processing jobs with configurable parameters
-
-**Job Configuration:**
-```yaml
-job:
-  job_id: UUID
-  job_type: enum [qc_only, alignment, full_pipeline, off_target_only]
-  priority: enum [low, normal, high, urgent]
-  input_files: array[string]
-  reference_genome: string
-  parameters: object
-  notification_email: string (optional)
-```
-
-#### FR-2.3.2: Job Status Tracking
-**Priority:** High
-**Description:** Real-time job status monitoring
-
-**Status States:**
-- `queued`: Job submitted, awaiting resources
-- `running`: Active processing
-- `completed`: Successful completion
-- `failed`: Error occurred (with error details)
-- `cancelled`: User-initiated cancellation
-
-**Acceptance Criteria:**
-- Status updates every 30 seconds
-- Progress percentage estimation
-- ETA calculation based on file size
-- Resource usage metrics (CPU, memory, disk I/O)
-
-#### FR-2.3.3: Job Cancellation
-**Priority:** Medium
-**Description:** Allow users to cancel running jobs
-
-**Requirements:**
-- Graceful shutdown of running processes
-- Cleanup of partial output files
-- Preserve logs for debugging
-- Immediate status update to `cancelled`
-
-### 2.4 Results Retrieval (FR-2.4)
-
-#### FR-2.4.1: Experiment Results API
-**Priority:** High
-**Description:** Retrieve processed results for completed experiments
-
-**Response Structure:**
-```json
-{
-  "experiment_id": "uuid",
-  "status": "completed",
-  "qc_metrics": {
-    "total_reads": 45000000,
-    "aligned_reads": 42300000,
-    "alignment_rate": 0.94,
-    "mean_quality": 36.5
-  },
-  "target_analysis": {
-    "knockdown_efficiency": 0.87,
-    "p_value": 1.2e-15,
-    "tpm_control": 125.3,
-    "tpm_treated": 16.2
-  },
-  "off_targets": [
-    {
-      "rank": 1,
-      "gene": "GAPDH",
-      "transcript_id": "ENST00000229239",
-      "score": 0.82,
-      "mismatches": 2,
-      "position": "chr12:6646711-6646733"
-    }
-  ],
-  "immune_response": {
-    "interferon_score": 2.35,
-    "activated_pathways": ["Type_I_IFN", "RIG-I"],
-    "sig_genes": 47
-  },
-  "files": {
-    "alignment_bam": "s3://results/exp123/aligned.bam",
-    "qc_report": "s3://results/exp123/qc_report.html",
-    "expression_matrix": "s3://results/exp123/expression.tsv"
-  }
-}
-```
-
-#### FR-2.4.2: File Downloads
-**Priority:** High
-**Description:** Secure download links for result files
-
-**Supported Formats:**
-- BAM/BAI (alignment files)
-- VCF (variant calls)
-- TSV (expression matrices)
-- HTML (QC reports)
-- BED (off-target coordinates)
-
-**Security:**
-- Pre-signed URLs with 1-hour expiration
-- User authentication required
-- Audit logging of all downloads
-
-### 2.5 Real-Time Monitoring (FR-2.5)
-
-#### FR-2.5.1: WebSocket Job Updates
-**Priority:** Medium
-**Description:** Real-time job progress via WebSocket connection
-
-**Message Format:**
-```json
-{
-  "job_id": "uuid",
-  "timestamp": "2025-10-12T00:15:30Z",
-  "event": "progress_update",
-  "data": {
-    "stage": "alignment",
-    "progress": 0.65,
-    "eta_seconds": 420,
-    "message": "Processing chromosome 7..."
-  }
-}
-```
-
-**Events:**
-- `job_started`
-- `progress_update`
-- `stage_complete`
-- `job_completed`
-- `job_failed`
-- `warning_message`
-
----
-
-## 3. Non-Functional Requirements
-
-### 3.1 Performance (NFR-3.1)
-
-#### NFR-3.1.1: Throughput
-**Description:** System shall process multiple experiments concurrently
-
-**Requirements:**
-- 10 concurrent full-pipeline jobs
-- 50 concurrent QC-only jobs
-- Queue management with priority scheduling
-
-#### NFR-3.1.2: Processing Time
-**Description:** Maximum processing time per experiment
-
-**Benchmarks:**
-- QC analysis: <15 minutes (30M reads)
-- Alignment: <2 hours (30M paired-end reads)
-- Off-target prediction: <30 minutes
-- Full pipeline: <3 hours (standard experiment)
-
-#### NFR-3.1.3: API Response Time
-**Description:** API latency requirements
-
-**Targets:**
-- Status queries: <200ms (p95)
-- Results retrieval: <500ms (p95)
-- Job submission: <1s (p95)
-- WebSocket latency: <100ms
-
-### 3.2 Scalability (NFR-3.2)
-
-#### NFR-3.2.1: Data Volume
-**Description:** Support for large-scale studies
-
-**Capacity:**
-- 10,000 experiments per year
-- 500TB total storage (3-year retention)
-- 1PB genomic reference data
-
-#### NFR-3.2.2: Concurrent Users
-**Description:** Multi-user support
-
-**Requirements:**
-- 100 concurrent API users
-- 500 authenticated sessions
-- Role-based access control (RBAC)
-
-### 3.3 Reliability (NFR-3.3)
-
-#### NFR-3.3.1: Uptime
-**Description:** System availability SLA
-
-**Target:** 99.5% uptime (43 hours downtime/year)
-
-**Measures:**
-- Automated health checks
-- Database replication
-- Graceful degradation
-- Automatic job retry (transient failures)
-
-#### NFR-3.3.2: Data Integrity
-**Description:** Prevent data loss and corruption
-
-**Requirements:**
-- Daily automated backups
-- Point-in-time recovery (7 days)
-- Checksum validation for all files
-- Transaction-based database operations
-
-### 3.4 Security (NFR-3.4)
-
-#### NFR-3.4.1: Authentication
-**Description:** Secure user authentication
-
-**Requirements:**
-- OAuth2/OpenID Connect support
-- Multi-factor authentication (MFA)
-- API key authentication for programmatic access
-- Session timeout: 24 hours
-
-#### NFR-3.4.2: Authorization
-**Description:** Fine-grained access control
-
-**Roles:**
-- `admin`: Full system access
-- `researcher`: Submit jobs, view own results
-- `analyst`: Read-only access to all data
-- `guest`: View public datasets only
-
-#### NFR-3.4.3: Data Encryption
-**Description:** Protect sensitive data
-
-**Requirements:**
-- TLS 1.3 for all API traffic
-- AES-256 encryption at rest
-- Encrypted database connections
-- Secure key management (AWS KMS/HashiCorp Vault)
-
-#### NFR-3.4.4: Audit Logging
-**Description:** Complete audit trail
-
-**Log Events:**
-- User authentication/authorization
-- Job submissions/cancellations
-- Data access/downloads
-- Configuration changes
-- System errors
-
-**Retention:** 1 year
-
-### 3.5 Compliance (NFR-3.5)
-
-#### NFR-3.5.1: HIPAA Compliance
-**Description:** If processing human samples
-
-**Requirements:**
-- PHI de-identification
-- Business Associate Agreement (BAA)
-- Encrypted storage
-- Access audit logs
-
-#### NFR-3.5.2: GDPR Compliance
-**Description:** EU data protection
-
-**Requirements:**
-- Right to access personal data
-- Right to deletion
-- Data portability
-- Consent management
-
-#### NFR-3.5.3: NIH Data Sharing
-**Description:** Compliance with NIH genomic data sharing policy
-
-**Requirements:**
-- Data submission to dbGaP (if required)
-- Institutional Data Access Committee (DAC)
-- Data Use Agreements (DUA)
-
----
-
-## 4. Data Provenance Requirements
-
-### 4.1 Provenance Tracking (PR-4.1)
-
-#### PR-4.1.1: W3C PROV-O Compliance
-**Description:** Implement provenance using W3C PROV ontology
-
-**Entities:**
-- Raw FASTQ files
-- Reference genomes
-- Aligned BAM files
-- Expression matrices
-- Off-target predictions
-
-**Activities:**
-- Quality control
-- Read alignment
-- Quantification
-- Statistical analysis
-
-**Agents:**
-- Software tools (versions)
-- User accounts
-- Automated pipelines
-
-#### PR-4.1.2: Provenance Graph
-**Description:** Complete lineage tracking
-
-**Requirements:**
-- Directed acyclic graph (DAG) representation
-- RDF serialization (Turtle/JSON-LD)
-- Queryable via SPARQL
-- Visualization in web interface
-
-**Example Provenance Chain:**
-```
-raw_fastq.gz → [FastQC] → qc_report.html
-             → [Trimmomatic] → trimmed_fastq.gz
-             → [STAR] → aligned.bam
-             → [featureCounts] → expression.tsv
-             → [DESeq2] → differential_expression.csv
-```
-
-#### PR-4.1.3: Tool Version Tracking
-**Description:** Record exact software versions
-
-**Required Metadata:**
-```yaml
-tool_provenance:
-  tool_name: STAR
-  version: 2.7.10b
-  parameters:
-    genomeDir: /ref/rheMac10_star
-    outSAMtype: BAM SortedByCoordinate
-    quantMode: TranscriptomeSAM
-  docker_image: quay.io/biocontainers/star:2.7.10b
-  execution_time: 7234 seconds
-  exit_code: 0
-  stdout_log: s3://logs/job123/star_stdout.log
-  stderr_log: s3://logs/job123/star_stderr.log
-```
-
-### 4.2 Reproducibility (PR-4.2)
-
-#### PR-4.2.1: Containerized Workflows
-**Description:** All tools packaged in Docker containers
-
-**Requirements:**
-- Dockerfile for each tool
-- Fixed base image tags (no `latest`)
-- Multi-stage builds for optimization
-- Security scanning (Trivy/Snyk)
-
-#### PR-4.2.2: Workflow Definition
-**Description:** Declarative pipeline definition
-
-**Format:** Nextflow DSL2 or Snakemake
-
-**Example:**
-```nextflow
-process ALIGNMENT {
-  container 'quay.io/biocontainers/star:2.7.10b'
-
-  input:
-    tuple val(sample_id), path(reads)
-    path(genome_index)
-
-  output:
-    tuple val(sample_id), path("${sample_id}.bam")
-
-  script:
-    """
-    STAR --genomeDir ${genome_index} \
-         --readFilesIn ${reads} \
-         --outSAMtype BAM SortedByCoordinate \
-         --runThreadN ${task.cpus}
-    """
-}
-```
-
----
-
-## 5. System Constraints
-
-### 5.1 Technical Constraints
-
-#### C-5.1.1: Infrastructure
-- **Platform:** Linux (Ubuntu 22.04 LTS)
-- **Container Runtime:** Docker 24.0+
-- **Orchestration:** Kubernetes 1.28+ (optional)
-- **Database:** PostgreSQL 15+, MongoDB 7.0+
-- **Object Storage:** S3-compatible (AWS/MinIO)
-
-#### C-5.1.2: Programming Languages
-- **Backend API:** Python 3.11+ (FastAPI)
-- **Pipeline:** Nextflow 23.10+ or Snakemake 7.32+
-- **Bioinformatics Tools:** STAR, HISAT2, featureCounts, DESeq2, BLAST+
-
-#### C-5.1.3: Dependencies
-- Minimum 32GB RAM for alignment jobs
-- 8 CPU cores per concurrent job
-- 10TB NVMe storage for temporary files
-- 1Gbps network connectivity
-
-### 5.2 Business Constraints
-
-#### C-5.2.1: Budget
-- Development: $150,000 (6 months)
-- Infrastructure: $5,000/month (cloud costs)
-- Maintenance: 1 FTE DevOps engineer
-
-#### C-5.2.2: Timeline
-- **Phase 1** (Months 1-2): Core pipeline + API
-- **Phase 2** (Months 3-4): Off-target prediction + immune analysis
-- **Phase 3** (Months 5-6): Provenance + production deployment
-
-#### C-5.2.3: Team
-- 1 Bioinformatics Lead
-- 2 Software Engineers
-- 1 DevOps Engineer
-- 1 QA Analyst
-
-### 5.3 Regulatory Constraints
-
-#### C-5.3.1: Animal Research
-- IACUC approval required
-- Compliance with USDA Animal Welfare Act
-- Detailed experimental protocols
-
-#### C-5.3.2: Data Sharing
-- NIH Genomic Data Sharing Policy
-- Institutional Data Sharing Agreement
-- dbGaP submission (if human-relevant)
-
----
-
-## 6. Acceptance Criteria Summary
-
-### Phase 1: Core Functionality
-- [ ] FASTQ upload with validation (FR-2.1.1)
-- [ ] QC analysis pipeline (FR-2.2.1)
-- [ ] Read alignment (FR-2.2.2)
-- [ ] Job submission API (FR-2.3.1)
-- [ ] Status tracking (FR-2.3.2)
-- [ ] PostgreSQL schema implemented
-- [ ] MongoDB collections created
-- [ ] API response time <500ms (NFR-3.1.3)
-
-### Phase 2: Advanced Analysis
-- [ ] Target site identification (FR-2.2.3)
-- [ ] Off-target prediction (FR-2.2.4)
-- [ ] Immune response analysis (FR-2.2.5)
-- [ ] Results retrieval API (FR-2.4.1)
-- [ ] WebSocket monitoring (FR-2.5.1)
-- [ ] Processing time <3 hours (NFR-3.1.2)
-
-### Phase 3: Production Readiness
-- [ ] Provenance tracking (PR-4.1)
-- [ ] Containerized workflows (PR-4.2.1)
-- [ ] Authentication/authorization (NFR-3.4.1/3.4.2)
-- [ ] Audit logging (NFR-3.4.4)
-- [ ] 99.5% uptime (NFR-3.3.1)
-- [ ] Automated backups (NFR-3.3.2)
-- [ ] HIPAA compliance (if required) (NFR-3.5.1)
-
----
-
-## 7. Validation & Testing
-
-### 7.1 Unit Testing
-- Minimum 90% code coverage
-- Pytest for Python components
-- Mock external services (S3, databases)
-
-### 7.2 Integration Testing
-- End-to-end pipeline tests with sample data
-- API contract testing (Postman/Newman)
-- Database migration testing
-
-### 7.3 Performance Testing
-- Load testing: 100 concurrent users (Locust)
-- Stress testing: Maximum job throughput
-- Latency benchmarks: API response times
-
-### 7.4 Validation Data
-- **Positive Control:** Known efficient guide RNA (e.g., KRAS-targeting)
-- **Negative Control:** Non-targeting scrambled guide RNA
-- **Reference Dataset:** Public CRISPR-Cas13 dataset (SRA)
-
----
-
-## 8. Glossary
-
-| Term | Definition |
-|------|------------|
-| **BAM** | Binary Alignment/Map format for aligned sequencing reads |
-| **Cas13** | RNA-guided RNA-targeting CRISPR effector protein |
-| **dbGaP** | Database of Genotypes and Phenotypes (NCBI) |
-| **FASTQ** | Text-based format for storing nucleotide sequences with quality scores |
-| **GA4GH** | Global Alliance for Genomics and Health |
-| **gRNA** | Guide RNA that directs Cas13 to target transcript |
-| **IACUC** | Institutional Animal Care and Use Committee |
-| **ISG** | Interferon-Stimulated Gene |
-| **NGS** | Next-Generation Sequencing |
-| **PROV-O** | W3C Provenance Ontology |
-| **RPKM** | Reads Per Kilobase per Million mapped reads |
-| **TPM** | Transcripts Per Million (normalized expression metric) |
-
----
-
-## 9. References
-
-1. Abudayyeh, O. O., et al. (2017). "C2c2 is a single-component programmable RNA-guided RNA-targeting CRISPR effector." *Science*, 353(6299).
-2. GA4GH Data Working Group. (2023). "GA4GH Data Standards." https://www.ga4gh.org/
-3. NCBI. (2023). "Sequence Read Archive Format Guide." https://www.ncbi.nlm.nih.gov/sra/
-4. W3C. (2013). "PROV-O: The PROV Ontology." https://www.w3.org/TR/prov-o/
-5. NIH. (2023). "Genomic Data Sharing Policy." https://sharing.nih.gov/genomic-data-sharing-policy
-
----
-
-**Document Control:**
-- **Author:** SPARC Specification Agent
-- **Reviewers:** Bioinformatics Lead, Software Architect
-- **Next Review Date:** 2025-11-12
-- **Version History:**
-  - v1.0.0 (2025-10-12): Initial specification