@saibolla/ada 0.1.2

package/skills/exploratory-data-analysis/references/bioinformatics_genomics_formats.md

@@ -0,0 +1,664 @@
+# Bioinformatics and Genomics File Formats Reference
+
+This reference covers file formats used in genomics, transcriptomics, sequence analysis, and related bioinformatics applications.
+
+## Sequence Data Formats
+
+### .fasta / .fa / .fna - FASTA Format
+**Description:** Text-based format for nucleotide or protein sequences
+**Typical Data:** DNA, RNA, or protein sequences with headers
+**Use Cases:** Sequence storage, BLAST searches, alignments
+**Python Libraries:**
+- `Biopython`: `SeqIO.parse('file.fasta', 'fasta')`
+- `pyfaidx`: Fast indexed FASTA access
+- `screed`: Fast sequence parsing
+**EDA Approach:**
+- Sequence count and length distribution
+- GC content analysis
+- N content (ambiguous bases)
+- Sequence ID parsing
+- Duplicate detection
+- Quality metrics for assemblies (N50, L50)
+
+### .fastq / .fq - FASTQ Format
+**Description:** Sequence data with base quality scores
+**Typical Data:** Raw sequencing reads with Phred quality scores
+**Use Cases:** NGS data, quality control, read mapping
+**Python Libraries:**
+- `Biopython`: `SeqIO.parse('file.fastq', 'fastq')`
+- `pysam`: Fast FASTQ/BAM operations
+- `HTSeq`: Sequencing data analysis
+**EDA Approach:**
+- Read count and length distribution
+- Quality score distribution (per-base, per-read)
+- GC content and bias
+- Duplicate rate estimation
+- Adapter contamination detection
+- k-mer frequency analysis
+- Encoding format validation (Phred33/64)
+
+### .sam - Sequence Alignment/Map
+**Description:** Tab-delimited text format for alignments
+**Typical Data:** Aligned sequencing reads with mapping quality
+**Use Cases:** Read alignment storage, variant calling
+**Python Libraries:**
+- `pysam`: `pysam.AlignmentFile('file.sam', 'r')`
+- `HTSeq`: `HTSeq.SAM_Reader('file.sam')`
+**EDA Approach:**
+- Mapping rate and quality distribution
+- Coverage analysis
+- Insert size distribution (paired-end)
+- Alignment flags distribution
+- CIGAR string patterns
+- Mismatch and indel rates
+- Duplicate and supplementary alignment counts
+
+### .bam - Binary Alignment/Map
+**Description:** Compressed binary version of SAM
+**Typical Data:** Aligned reads in compressed format
+**Use Cases:** Efficient storage and processing of alignments
+**Python Libraries:**
+- `pysam`: Full BAM support with indexing
+- `bamnostic`: Pure Python BAM reader
+**EDA Approach:**
+- Same as SAM plus:
+- Compression ratio analysis
+- Index file (.bai) validation
+- Chromosome-wise statistics
+- Strand bias detection
+- Read group analysis
+
+### .cram - CRAM Format
+**Description:** Highly compressed alignment format
+**Typical Data:** Reference-compressed aligned reads
+**Use Cases:** Long-term storage, space-efficient archives
+**Python Libraries:**
+- `pysam`: CRAM support (requires reference)
+- Reference genome must be accessible
+**EDA Approach:**
+- Compression efficiency vs BAM
+- Reference dependency validation
+- Lossy vs lossless compression assessment
+- Decompression performance
+- Similar alignment metrics as BAM
+
+### .bed - Browser Extensible Data
+**Description:** Tab-delimited format for genomic features
+**Typical Data:** Genomic intervals (chr, start, end) with annotations
+**Use Cases:** Peak calling, variant annotation, genome browsing
+**Python Libraries:**
+- `pybedtools`: `pybedtools.BedTool('file.bed')`
+- `pyranges`: `pyranges.read_bed('file.bed')`
+- `pandas`: Simple BED reading
+**EDA Approach:**
+- Feature count and size distribution
+- Chromosome distribution
+- Strand bias
+- Score distribution (if present)
+- Overlap and proximity analysis
+- Coverage statistics
+- Gap analysis between features
+
+### .bedGraph - BED with Graph Data
+**Description:** BED format with per-base signal values
+**Typical Data:** Continuous-valued genomic data (coverage, signals)
+**Use Cases:** Coverage tracks, ChIP-seq signals, methylation
+**Python Libraries:**
+- `pyBigWig`: Can convert to bigWig
+- `pybedtools`: BedGraph operations
+**EDA Approach:**
+- Signal distribution statistics
+- Genome coverage percentage
+- Signal dynamics (peaks, valleys)
+- Chromosome-wise signal patterns
+- Quantile analysis
+- Zero-coverage regions
+
+### .bigWig / .bw - Binary BigWig
+**Description:** Indexed binary format for genome-wide signal data
+**Typical Data:** Continuous genomic signals (compressed and indexed)
+**Use Cases:** Efficient genome browser tracks, large-scale data
+**Python Libraries:**
+- `pyBigWig`: `pyBigWig.open('file.bw')`
+- `pybbi`: BigWig/BigBed interface
+**EDA Approach:**
+- Signal statistics extraction
+- Zoom level analysis
+- Regional signal extraction
+- Efficient genome-wide summaries
+- Compression efficiency
+- Index structure analysis
+
+### .bigBed / .bb - Binary BigBed
+**Description:** Indexed binary BED format
+**Typical Data:** Genomic features (compressed and indexed)
+**Use Cases:** Large feature sets, genome browsers
+**Python Libraries:**
+- `pybbi`: BigBed reading
+- `pybigtools`: Modern BigBed interface
+**EDA Approach:**
+- Feature density analysis
+- Efficient interval queries
+- Zoom level validation
+- Index performance metrics
+- Feature size statistics
+
+### .gff / .gff3 - General Feature Format
+**Description:** Tab-delimited format for genomic annotations
+**Typical Data:** Gene models, transcripts, exons, regulatory elements
+**Use Cases:** Genome annotation, gene prediction
+**Python Libraries:**
+- `BCBio.GFF`: Biopython GFF module
+- `gffutils`: `gffutils.create_db('file.gff3')`
+- `pyranges`: GFF support
+**EDA Approach:**
+- Feature type distribution (gene, exon, CDS, etc.)
+- Gene structure validation
+- Strand balance
+- Hierarchical relationship validation
+- Phase validation for CDS
+- Attribute completeness
+- Gene model statistics (introns, exons per gene)
+
+### .gtf - Gene Transfer Format
+**Description:** GFF2-based format for gene annotations
+**Typical Data:** Gene and transcript annotations
+**Use Cases:** RNA-seq analysis, gene quantification
+**Python Libraries:**
+- `pyranges`: `pyranges.read_gtf('file.gtf')`
+- `gffutils`: GTF database creation
+- `HTSeq`: GTF reading for counts
+**EDA Approach:**
+- Transcript isoform analysis
+- Gene structure completeness
+- Exon number distribution
+- Transcript length distribution
+- TSS and TES analysis
+- Biotype distribution
+- Overlapping gene detection
+
+### .vcf - Variant Call Format
+**Description:** Text format for genetic variants
+**Typical Data:** SNPs, indels, structural variants with annotations
+**Use Cases:** Variant calling, population genetics, GWAS
+**Python Libraries:**
+- `pysam`: `pysam.VariantFile('file.vcf')`
+- `cyvcf2`: Fast VCF parsing
+- `PyVCF`: Older but comprehensive
+**EDA Approach:**
+- Variant count by type (SNP, indel, SV)
+- Quality score distribution
+- Allele frequency spectrum
+- Transition/transversion ratio
+- Heterozygosity rates
+- Missing genotype analysis
+- Hardy-Weinberg equilibrium
+- Annotation completeness (if annotated)
+
+### .bcf - Binary VCF
+**Description:** Compressed binary variant format
+**Typical Data:** Same as VCF but binary
+**Use Cases:** Efficient variant storage and processing
+**Python Libraries:**
+- `pysam`: Full BCF support
+- `cyvcf2`: Optimized BCF reading
+**EDA Approach:**
+- Same as VCF plus:
+- Compression efficiency
+- Indexing validation
+- Read performance metrics
+
+### .gvcf - Genomic VCF
+**Description:** VCF with reference confidence blocks
+**Typical Data:** All positions (variant and non-variant)
+**Use Cases:** Joint genotyping workflows, GATK
+**Python Libraries:**
+- `pysam`: GVCF support
+- Standard VCF parsers
+**EDA Approach:**
+- Reference block analysis
+- Coverage uniformity
+- Variant density
+- Genotype quality across genome
+- Reference confidence distribution
+
+## RNA-Seq and Expression Data
+
+### .counts - Gene Count Matrix
+**Description:** Tab-delimited gene expression counts
+**Typical Data:** Gene IDs with read counts per sample
+**Use Cases:** RNA-seq quantification, differential expression
+**Python Libraries:**
+- `pandas`: `pd.read_csv('file.counts', sep='\t')`
+- `scanpy` (for single-cell): `sc.read_csv()`
+**EDA Approach:**
+- Library size distribution
+- Detection rate (genes per sample)
+- Zero-inflation analysis
+- Count distribution (log scale)
+- Outlier sample detection
+- Correlation between replicates
+- PCA for sample relationships
+
+### .tpm / .fpkm - Normalized Expression
+**Description:** Normalized gene expression values
+**Typical Data:** TPM (transcripts per million) or FPKM values
+**Use Cases:** Cross-sample comparison, visualization
+**Python Libraries:**
+- `pandas`: Standard CSV reading
+- `anndata`: For integrated analysis
+**EDA Approach:**
+- Expression distribution
+- Highly expressed gene identification
+- Sample clustering
+- Batch effect detection
+- Coefficient of variation analysis
+- Dynamic range assessment
+
+### .mtx - Matrix Market Format
+**Description:** Sparse matrix format (common in single-cell)
+**Typical Data:** Sparse count matrices (cells × genes)
+**Use Cases:** Single-cell RNA-seq, large sparse matrices
+**Python Libraries:**
+- `scipy.io`: `scipy.io.mmread('file.mtx')`
+- `scanpy`: `sc.read_mtx('file.mtx')`
+**EDA Approach:**
+- Sparsity analysis
+- Cell and gene filtering thresholds
+- Doublet detection metrics
+- Mitochondrial fraction
+- UMI count distribution
+- Gene detection per cell
+
+### .h5ad - Anndata Format
+**Description:** HDF5-based annotated data matrix
+**Typical Data:** Expression matrix with metadata (cells, genes)
+**Use Cases:** Single-cell RNA-seq analysis with Scanpy
+**Python Libraries:**
+- `scanpy`: `sc.read_h5ad('file.h5ad')`
+- `anndata`: Direct AnnData manipulation
+**EDA Approach:**
+- Cell and gene counts
+- Metadata completeness
+- Layer availability (raw, normalized)
+- Embedding presence (PCA, UMAP)
+- QC metrics distribution
+- Batch information
+- Cell type annotation coverage
+
+### .loom - Loom Format
+**Description:** HDF5-based format for omics data
+**Typical Data:** Expression matrices with metadata
+**Use Cases:** Single-cell data, RNA velocity analysis
+**Python Libraries:**
+- `loompy`: `loompy.connect('file.loom')`
+- `scanpy`: Can import loom files
+**EDA Approach:**
+- Layer analysis (spliced, unspliced)
+- Row and column attribute exploration
+- Graph connectivity analysis
+- Cluster assignments
+- Velocity-specific metrics
+
+### .rds - R Data Serialization
+**Description:** R object storage (often Seurat objects)
+**Typical Data:** R analysis results, especially single-cell
+**Use Cases:** R-Python data exchange
+**Python Libraries:**
+- `pyreadr`: `pyreadr.read_r('file.rds')`
+- `rpy2`: For full R integration
+- Conversion tools to AnnData
+**EDA Approach:**
+- Object type identification
+- Data structure exploration
+- Metadata extraction
+- Conversion validation
+
+## Alignment and Assembly Formats
+
+### .maf - Multiple Alignment Format
+**Description:** Text format for multiple sequence alignments
+**Typical Data:** Genome-wide or local multiple alignments
+**Use Cases:** Comparative genomics, conservation analysis
+**Python Libraries:**
+- `Biopython`: `AlignIO.parse('file.maf', 'maf')`
+- `bx-python`: MAF-specific tools
+**EDA Approach:**
+- Alignment block statistics
+- Species coverage
+- Gap analysis
+- Conservation scoring
+- Alignment quality metrics
+- Block length distribution
+
+### .axt - Pairwise Alignment Format
+**Description:** Pairwise alignment format (UCSC)
+**Typical Data:** Pairwise genomic alignments
+**Use Cases:** Genome comparison, synteny analysis
+**Python Libraries:**
+- Custom parsers (simple format)
+- `bx-python`: AXT support
+**EDA Approach:**
+- Alignment score distribution
+- Identity percentage
+- Syntenic block identification
+- Gap size analysis
+- Coverage statistics
+
+### .chain - Chain Alignment Format
+**Description:** Genome coordinate mapping chains
+**Typical Data:** Coordinate transformations between genome builds
+**Use Cases:** Liftover, coordinate conversion
+**Python Libraries:**
+- `pyliftover`: Chain file usage
+- Custom parsers for chain format
+**EDA Approach:**
+- Chain score distribution
+- Coverage of source genome
+- Gap analysis
+- Inversion detection
+- Mapping quality assessment
+
+### .psl - Pattern Space Layout
+**Description:** BLAT/BLAST alignment format
+**Typical Data:** Alignment results from BLAT
+**Use Cases:** Transcript mapping, similarity searches
+**Python Libraries:**
+- Custom parsers (tab-delimited)
+- `pybedtools`: Can handle PSL
+**EDA Approach:**
+- Match percentage distribution
+- Gap statistics
+- Query coverage
+- Multiple mapping analysis
+- Alignment quality metrics
+
+## Genome Assembly and Annotation
+
+### .agp - Assembly Golden Path
+**Description:** Assembly structure description
+**Typical Data:** Scaffold composition, gap information
+**Use Cases:** Genome assembly representation
+**Python Libraries:**
+- Custom parsers (simple tab-delimited)
+- Assembly analysis tools
+**EDA Approach:**
+- Scaffold statistics (N50, L50)
+- Gap type and size distribution
+- Component length analysis
+- Assembly contiguity metrics
+- Unplaced contig analysis
+
+### .scaffolds / .contigs - Assembly Sequences
+**Description:** Assembled sequences (usually FASTA)
+**Typical Data:** Assembled genomic sequences
+**Use Cases:** Genome assembly output
+**Python Libraries:**
+- Same as FASTA format
+- Assembly-specific tools (QUAST)
+**EDA Approach:**
+- Assembly statistics (N50, N90, etc.)
+- Length distribution
+- Coverage analysis
+- Gap (N) content
+- Duplication assessment
+- BUSCO completeness (if annotations available)
+
+### .2bit - Compressed Genome Format
+**Description:** UCSC compact genome format
+**Typical Data:** Reference genomes (highly compressed)
+**Use Cases:** Efficient genome storage and access
+**Python Libraries:**
+- `py2bit`: `py2bit.open('file.2bit')`
+- `twobitreader`: Alternative reader
+**EDA Approach:**
+- Compression efficiency
+- Random access performance
+- Sequence extraction validation
+- Masked region analysis
+- N content and distribution
+
+### .sizes - Chromosome Sizes
+**Description:** Simple format with chromosome lengths
+**Typical Data:** Tab-delimited chromosome names and sizes
+**Use Cases:** Genome browsers, coordinate validation
+**Python Libraries:**
+- Simple file reading with pandas
+- Built into many genomic tools
+**EDA Approach:**
+- Genome size calculation
+- Chromosome count
+- Size distribution
+- Karyotype validation
+- Completeness check against reference
+
+## Phylogenetics and Evolution
+
+### .nwk / .newick - Newick Tree Format
+**Description:** Parenthetical tree representation
+**Typical Data:** Phylogenetic trees with branch lengths
+**Use Cases:** Evolutionary analysis, tree visualization
+**Python Libraries:**
+- `Biopython`: `Phylo.read('file.nwk', 'newick')`
+- `ete3`: `ete3.Tree('file.nwk')`
+- `dendropy`: Phylogenetic computing
+**EDA Approach:**
+- Tree structure analysis (tips, internal nodes)
+- Branch length distribution
+- Tree balance metrics
+- Ultrametricity check
+- Bootstrap support analysis
+- Topology validation
+
+### .nexus - Nexus Format
+**Description:** Rich format for phylogenetic data
+**Typical Data:** Alignments, trees, character matrices
+**Use Cases:** Phylogenetic software interchange
+**Python Libraries:**
+- `Biopython`: Nexus support
+- `dendropy`: Comprehensive Nexus handling
+**EDA Approach:**
+- Data block analysis
+- Character type distribution
+- Tree block validation
+- Taxa consistency
+- Command block parsing
+- Format compliance checking
+
+### .phylip - PHYLIP Format
+**Description:** Sequence alignment format (strict/relaxed)
+**Typical Data:** Multiple sequence alignments
+**Use Cases:** Phylogenetic analysis input
+**Python Libraries:**
+- `Biopython`: `AlignIO.read('file.phy', 'phylip')`
+- `dendropy`: PHYLIP support
+**EDA Approach:**
+- Alignment dimensions
+- Sequence length uniformity
+- Gap position analysis
+- Informative site calculation
+- Format variant detection (strict vs relaxed)
+
+### .paml - PAML Output
+**Description:** Output from PAML phylogenetic software
+**Typical Data:** Evolutionary model results, dN/dS ratios
+**Use Cases:** Molecular evolution analysis
+**Python Libraries:**
+- Custom parsers for specific PAML programs
+- `Biopython`: Basic PAML parsing
+**EDA Approach:**
+- Model parameter extraction
+- Likelihood values
+- dN/dS ratio distribution
+- Branch-specific results
+- Convergence assessment
+
+## Protein and Structure Data
+
+### .embl - EMBL Format
+**Description:** Rich sequence annotation format
+**Typical Data:** Sequences with extensive annotations
+**Use Cases:** Sequence databases, genome records
+**Python Libraries:**
+- `Biopython`: `SeqIO.read('file.embl', 'embl')`
+**EDA Approach:**
+- Feature annotation completeness
+- Sequence length and type
+- Reference information
+- Cross-reference validation
+- Feature overlap analysis
+
+### .genbank / .gb / .gbk - GenBank Format
+**Description:** NCBI's sequence annotation format
+**Typical Data:** Annotated sequences with features
+**Use Cases:** Sequence databases, annotation transfer
+**Python Libraries:**
+- `Biopython`: `SeqIO.parse('file.gb', 'genbank')`
+**EDA Approach:**
+- Feature type distribution
+- CDS analysis (start codons, stops)
+- Translation validation
+- Annotation completeness
+- Source organism extraction
+- Reference and publication info
+- Locus tag consistency
+
+### .sff - Standard Flowgram Format
+**Description:** 454/Roche sequencing data format
+**Typical Data:** Raw pyrosequencing flowgrams
+**Use Cases:** Legacy 454 sequencing data
+**Python Libraries:**
+- `Biopython`: `SeqIO.parse('file.sff', 'sff')`
+- Platform-specific tools
+**EDA Approach:**
+- Read count and length
+- Flowgram signal quality
+- Key sequence detection
+- Adapter trimming validation
+- Quality score distribution
+
+### .hdf5 (Genomics Specific)
+**Description:** HDF5 for genomics (10X, Hi-C, etc.)
+**Typical Data:** High-throughput genomics data
+**Use Cases:** 10X Genomics, spatial transcriptomics
+**Python Libraries:**
+- `h5py`: Low-level access
+- `scanpy`: For 10X data
+- `cooler`: For Hi-C data
+**EDA Approach:**
+- Dataset structure exploration
+- Barcode statistics
+- UMI counting
+- Feature-barcode matrix analysis
+- Spatial coordinates (if applicable)
+
+### .cool / .mcool - Cooler Format
+**Description:** HDF5-based Hi-C contact matrices
+**Typical Data:** Chromatin interaction matrices
+**Use Cases:** 3D genome analysis, Hi-C data
+**Python Libraries:**
+- `cooler`: `cooler.Cooler('file.cool')`
+- `hicstraw`: For .hic format
+**EDA Approach:**
+- Resolution analysis
+- Contact matrix statistics
+- Distance decay curves
+- Compartment analysis
+- TAD boundary detection
+- Balance factor validation
+
+### .hic - Hi-C Binary Format
+**Description:** Juicer binary Hi-C format
+**Typical Data:** Multi-resolution Hi-C matrices
+**Use Cases:** Hi-C analysis with Juicer tools
+**Python Libraries:**
+- `hicstraw`: `hicstraw.HiCFile('file.hic')`
+- `straw`: C++ library with Python bindings
+**EDA Approach:**
+- Available resolutions
+- Normalization methods
+- Contact statistics
+- Chromosomal interactions
+- Quality metrics
+
+### .bw (ChIP-seq / ATAC-seq specific)
+**Description:** BigWig files for epigenomics
+**Typical Data:** Coverage or enrichment signals
+**Use Cases:** ChIP-seq, ATAC-seq, DNase-seq
+**Python Libraries:**
+- `pyBigWig`: Standard bigWig access
+**EDA Approach:**
+- Peak enrichment patterns
+- Background signal analysis
+- Sample correlation
+- Signal-to-noise ratio
+- Library complexity metrics
+
+### .narrowPeak / .broadPeak - ENCODE Peak Formats
+**Description:** BED-based formats for peaks
+**Typical Data:** Peak calls with scores and p-values
+**Use Cases:** ChIP-seq peak calling output
+**Python Libraries:**
+- `pybedtools`: BED-compatible
+- Custom parsers for peak-specific fields
+**EDA Approach:**
+- Peak count and width distribution
+- Signal value distribution
+- Q-value and p-value analysis
+- Peak summit analysis
+- Overlap with known features
+- Motif enrichment preparation
+
+### .wig - Wiggle Format
+**Description:** Dense continuous genomic data
+**Typical Data:** Coverage or signal tracks
+**Use Cases:** Genome browser visualization
+**Python Libraries:**
+- `pyBigWig`: Can convert to bigWig
+- Custom parsers for wiggle format
+**EDA Approach:**
+- Signal statistics
+- Coverage metrics
+- Format variant (fixedStep vs variableStep)
+- Span parameter analysis
+- Conversion efficiency to bigWig
+
+### .ab1 - Sanger Sequencing Trace
+**Description:** Binary chromatogram format
+**Typical Data:** Sanger sequencing traces
+**Use Cases:** Capillary sequencing validation
+**Python Libraries:**
+- `Biopython`: `SeqIO.read('file.ab1', 'abi')`
+- `tracy` tools: For quality assessment
+**EDA Approach:**
+- Base calling quality
+- Trace quality scores
+- Mixed base detection
+- Primer and vector detection
+- Read length and quality region
+- Heterozygosity detection
+
+### .scf - Standard Chromatogram Format
+**Description:** Sanger sequencing chromatogram
+**Typical Data:** Base calls and confidence values
+**Use Cases:** Sequencing trace analysis
+**Python Libraries:**
+- `Biopython`: SCF format support
+**EDA Approach:**
+- Similar to AB1 format
+- Quality score profiles
+- Peak height ratios
+- Signal-to-noise metrics
+
+### .idx - Index Files (Generic)
+**Description:** Index files for various formats
+**Typical Data:** Fast random access indices
+**Use Cases:** Efficient data access (BAM, VCF, etc.)
+**Python Libraries:**
+- Format-specific libraries handle indices
+- `pysam`: Auto-handles BAI, CSI indices
+**EDA Approach:**
+- Index completeness validation
+- Binning strategy analysis
+- Access performance metrics
+- Index size vs data size ratio