@bgicli/bgicli 2.1.0 → 2.2.0

package/data/skills/bio-genome-intervals-coverage-analysis/SKILL.md

@@ -0,0 +1,300 @@
+<!--
+# COPYRIGHT NOTICE
+# This file is part of the "Universal Biomedical Skills" project.
+# Copyright (c) 2026 MD BABU MIA, PhD <md.babu.mia@mssm.edu>
+# All Rights Reserved.
+#
+# This code is proprietary and confidential.
+# Unauthorized copying of this file, via any medium is strictly prohibited.
+#
+# Provenance: Authenticated by MD BABU MIA
+
+-->
+
+---
+name: bio-genome-intervals-coverage-analysis
+description: Calculate read depth and coverage across genomic intervals using bedtools genomecov and coverage. Generate bedGraph files, compute per-base depth, and summarize coverage statistics. Use when assessing sequencing depth, creating coverage tracks, or evaluating target capture efficiency.
+tool_type: mixed
+primary_tool: bedtools
+measurable_outcome: Execute skill workflow successfully with valid output within 15 minutes.
+allowed-tools:
+  - read_file
+  - run_shell_command
+---
+
+# Coverage Analysis
+
+Calculate coverage and depth across genomic regions using bedtools and pybedtools.
+
+## genomecov - Genome-wide Coverage
+
+### Per-base Coverage (bedGraph)
+
+```bash
+# Generate bedGraph from BAM (per-base depth)
+bedtools genomecov -ibam alignments.bam -bg > coverage.bedGraph
+
+# Include zero-coverage regions
+bedtools genomecov -ibam alignments.bam -bga > coverage_with_zeros.bedGraph
+
+# Split by strand
+bedtools genomecov -ibam alignments.bam -bg -strand + > plus_strand.bedGraph
+bedtools genomecov -ibam alignments.bam -bg -strand - > minus_strand.bedGraph
+
+# Scale by total reads (RPM normalization)
+TOTAL=$(samtools view -c alignments.bam)
+SCALE=$(echo "scale=10; 1000000/$TOTAL" | bc)
+bedtools genomecov -ibam alignments.bam -bg -scale $SCALE > normalized.bedGraph
+
+# Use only 5' end of reads
+bedtools genomecov -ibam alignments.bam -bg -5 > five_prime.bedGraph
+
+# Use only 3' end of reads
+bedtools genomecov -ibam alignments.bam -bg -3 > three_prime.bedGraph
+```
+
+### Coverage Histogram
+
+```bash
+# Genome-wide coverage histogram
+bedtools genomecov -ibam alignments.bam > coverage_hist.txt
+
+# Output format: chr, depth, bases_at_depth, chr_size, fraction
+# genome  0       1000000  10000000  0.1
+# genome  1       5000000  10000000  0.5
+# ...
+```
+
+### Coverage from BED
+
+```bash
+# Coverage from BED intervals
+bedtools genomecov -i regions.bed -g genome.txt -bg > coverage.bedGraph
+
+# BED must be sorted
+bedtools sort -i regions.bed | bedtools genomecov -i stdin -g genome.txt -bg > coverage.bedGraph
+```
+
+### Python
+
+```python
+import pybedtools
+
+# From BAM
+bam = pybedtools.BedTool('alignments.bam')
+coverage = bam.genome_coverage(bg=True)
+coverage.saveas('coverage.bedGraph')
+
+# With zeros
+coverage = bam.genome_coverage(bga=True)
+
+# Normalized
+coverage = bam.genome_coverage(bg=True, scale=0.001)
+
+# From BED
+bed = pybedtools.BedTool('regions.bed')
+coverage = bed.genome_coverage(bg=True, g='genome.txt')
+```
+
+## coverage - Coverage per Feature
+
+### Basic Coverage
+
+```bash
+# Calculate how much of each region in A is covered by B
+bedtools coverage -a targets.bed -b reads.bed > coverage_per_target.bed
+
+# Output adds 4 columns: overlaps, bases_covered, region_length, fraction
+# chr1  100  200  region1  5  50  100  0.5
+
+# From BAM
+bedtools coverage -a targets.bed -b alignments.bam > coverage.bed
+
+# Count only (no coverage calculation)
+bedtools coverage -a targets.bed -b reads.bed -counts > counts.bed
+```
+
+### Coverage Options
+
+```bash
+# Mean coverage per region
+bedtools coverage -a targets.bed -b alignments.bam -mean > mean_coverage.bed
+
+# Same strand only
+bedtools coverage -a targets.bed -b alignments.bam -s > same_strand.bed
+
+# Report depth at each position (histogram)
+bedtools coverage -a targets.bed -b alignments.bam -d > per_base.bed
+
+# Require minimum overlap
+bedtools coverage -a targets.bed -b reads.bed -f 0.5 > min_overlap.bed
+
+# Split alignments (for RNA-seq)
+bedtools coverage -a exons.bed -b alignments.bam -split > exon_coverage.bed
+```
+
+### Python
+
+```python
+import pybedtools
+
+a = pybedtools.BedTool('targets.bed')
+b = pybedtools.BedTool('alignments.bam')
+
+# Basic coverage
+result = a.coverage(b)
+
+# Mean coverage
+result = a.coverage(b, mean=True)
+
+# Counts only
+result = a.coverage(b, counts=True)
+
+# Per-base depth
+result = a.coverage(b, d=True)
+
+result.saveas('coverage.bed')
+```
+
+## multicov - Counts Across Multiple BAMs
+
+```bash
+# Count reads in regions across multiple samples
+bedtools multicov -bams sample1.bam sample2.bam sample3.bam -bed regions.bed > counts.txt
+
+# Require mapping quality
+bedtools multicov -bams sample1.bam sample2.bam -bed regions.bed -q 30 > counts.txt
+
+# Split alignments
+bedtools multicov -bams sample1.bam sample2.bam -bed regions.bed -s -split > counts.txt
+```
+
+## Calculate Coverage Statistics
+
+### Mean/Median Depth
+
+```python
+import pybedtools
+import pandas as pd
+import numpy as np
+
+# Load coverage BED (from bedtools coverage -d)
+bed = pybedtools.BedTool('per_base_coverage.bed')
+df = bed.to_dataframe()
+
+# Calculate stats per region
+stats = df.groupby(['chrom', 'start', 'end']).agg({
+    'score': ['mean', 'median', 'std', 'max']
+}).reset_index()
+
+print(stats)
+```
+
+### Coverage Distribution
+
+```python
+import pybedtools
+
+# Get coverage histogram
+bam = pybedtools.BedTool('alignments.bam')
+hist = bam.genome_coverage()
+
+# Parse histogram
+depths = []
+fractions = []
+for line in open(hist.fn):
+    fields = line.strip().split('\t')
+    if fields[0] == 'genome':
+        depths.append(int(fields[1]))
+        fractions.append(float(fields[4]))
+
+# Calculate metrics
+import numpy as np
+mean_depth = sum(d * f for d, f in zip(depths, fractions))
+print(f'Mean depth: {mean_depth:.1f}x')
+```
+
+## Common Patterns
+
+### Target Region Coverage Summary
+
+```bash
+# Get per-region coverage stats
+bedtools coverage -a targets.bed -b alignments.bam | \
+    awk -v OFS='\t' '{
+        mean = ($NF > 0) ? $5/$6 : 0;
+        print $1, $2, $3, $4, $7, mean
+    }' > summary.bed
+
+# Regions with low coverage
+bedtools coverage -a targets.bed -b alignments.bam | \
+    awk '$NF < 0.8' > low_coverage.bed
+```
+
+### Normalize to CPM (Counts Per Million)
+
+```python
+import pybedtools
+
+bam = pybedtools.BedTool('alignments.bam')
+
+# Get total reads
+import subprocess
+result = subprocess.run(['samtools', 'view', '-c', 'alignments.bam'],
+                        capture_output=True, text=True)
+total_reads = int(result.stdout.strip())
+
+# Generate CPM-normalized bedGraph
+scale_factor = 1000000 / total_reads
+coverage = bam.genome_coverage(bg=True, scale=scale_factor)
+coverage.saveas('cpm_normalized.bedGraph')
+```
+
+### Exon Coverage for RNA-seq
+
+```bash
+# Calculate coverage across exons (handling spliced reads)
+bedtools coverage -a exons.bed -b alignments.bam -split > exon_coverage.bed
+
+# Summarize by gene
+awk -v OFS='\t' '{
+    gene = $4; gsub(/_exon.*/, "", gene);
+    sum[gene] += $NF * ($3-$2);
+    len[gene] += $3-$2;
+}
+END {
+    for (g in sum) print g, sum[g]/len[g];
+}' exon_coverage.bed > gene_coverage.txt
+```
+
+## bedGraph Format
+
+```
+# bedGraph: chr, start, end, value (0-based coordinates)
+chr1    0       100     0
+chr1    100     200     5.5
+chr1    200     300     10.2
+chr1    300     400     3.1
+```
+
+## Key Parameters
+
+| Tool | Parameter | Description |
+|------|-----------|-------------|
+| genomecov -bg | bedGraph | Output bedGraph format |
+| genomecov -bga | bedGraph all | Include zero coverage |
+| genomecov -scale | Normalize | Scale values by factor |
+| coverage -mean | Mean | Report mean coverage |
+| coverage -d | Per-base | Report per-position depth |
+| coverage -counts | Count | Count overlaps only |
+| multicov -q | Quality | Minimum mapping quality |
+
+## Related Skills
+
+- bigwig-tracks - Convert bedGraph to bigWig
+- alignment-files/sam-bam-basics - BAM processing
+- interval-arithmetic - Intersect with regions
+- chip-seq/chipseq-visualization - Peak coverage analysis
+
+
+<!-- AUTHOR_SIGNATURE: 9a7f3c2e-MD-BABU-MIA-2026-MSSM-SECURE -->

package/data/skills/bio-genome-intervals-gtf-gff-handling/SKILL.md

@@ -0,0 +1,345 @@
+<!--
+# COPYRIGHT NOTICE
+# This file is part of the "Universal Biomedical Skills" project.
+# Copyright (c) 2026 MD BABU MIA, PhD <md.babu.mia@mssm.edu>
+# All Rights Reserved.
+#
+# This code is proprietary and confidential.
+# Unauthorized copying of this file, via any medium is strictly prohibited.
+#
+# Provenance: Authenticated by MD BABU MIA
+
+-->
+
+---
+name: bio-genome-intervals-gtf-gff-handling
+description: Parse, query, and convert GTF and GFF3 annotation files. Extract gene, transcript, and exon coordinates using gffread, gtfparse, and gffutils. Use when extracting specific features from gene annotations or converting between annotation formats.
+tool_type: mixed
+primary_tool: gffread
+measurable_outcome: Execute skill workflow successfully with valid output within 15 minutes.
+allowed-tools:
+  - read_file
+  - run_shell_command
+---
+
+# GTF/GFF Handling
+
+GTF and GFF3 are standard gene annotation formats. Both use 1-based coordinates.
+
+## Format Comparison
+
+| Feature | GTF | GFF3 |
+|---------|-----|------|
+| Coordinate system | 1-based, inclusive | 1-based, inclusive |
+| Hierarchy | Implicit (gene_id, transcript_id) | Explicit (Parent attribute) |
+| Attribute format | key "value"; | key=value; |
+| Comments | # | # |
+| Fasta sequences | Not standard | ##FASTA directive |
+
+## GTF Format
+
+```
+chr1	HAVANA	gene	11869	14409	.	+	.	gene_id "ENSG00000223972"; gene_name "DDX11L1";
+chr1	HAVANA	transcript	11869	14409	.	+	.	gene_id "ENSG00000223972"; transcript_id "ENST00000456328";
+chr1	HAVANA	exon	11869	12227	.	+	.	gene_id "ENSG00000223972"; transcript_id "ENST00000456328"; exon_number "1";
+```
+
+## GFF3 Format
+
+```
+chr1	HAVANA	gene	11869	14409	.	+	.	ID=ENSG00000223972;Name=DDX11L1
+chr1	HAVANA	mRNA	11869	14409	.	+	.	ID=ENST00000456328;Parent=ENSG00000223972
+chr1	HAVANA	exon	11869	12227	.	+	.	ID=exon1;Parent=ENST00000456328
+```
+
+## Parse GTF with gtfparse (Python)
+
+### Installation
+
+```bash
+pip install gtfparse
+```
+
+### Basic Parsing
+
+```python
+import gtfparse
+
+# Load entire GTF
+df = gtfparse.read_gtf('annotation.gtf')
+
+# View columns
+print(df.columns)
+# ['seqname', 'source', 'feature', 'start', 'end', 'score', 'strand', 'frame',
+#  'gene_id', 'transcript_id', 'gene_name', ...]
+
+# Filter by feature type
+genes = df[df['feature'] == 'gene']
+transcripts = df[df['feature'] == 'transcript']
+exons = df[df['feature'] == 'exon']
+
+# Get specific gene
+gene_df = df[df['gene_name'] == 'TP53']
+```
+
+### Extract Gene Coordinates
+
+```python
+import gtfparse
+
+df = gtfparse.read_gtf('annotation.gtf')
+
+# All genes
+genes = df[df['feature'] == 'gene'][['seqname', 'start', 'end', 'strand', 'gene_id', 'gene_name']]
+
+# Convert to BED format (0-based)
+genes_bed = genes.copy()
+genes_bed['start'] = genes_bed['start'] - 1  # GTF is 1-based, BED is 0-based
+genes_bed = genes_bed[['seqname', 'start', 'end', 'gene_name', 'gene_id', 'strand']]
+genes_bed.to_csv('genes.bed', sep='\t', header=False, index=False)
+```
+
+### Get Exons for Gene
+
+```python
+import gtfparse
+
+df = gtfparse.read_gtf('annotation.gtf')
+
+# Get all exons for TP53
+tp53_exons = df[(df['gene_name'] == 'TP53') & (df['feature'] == 'exon')]
+tp53_exons = tp53_exons[['seqname', 'start', 'end', 'transcript_id', 'exon_number']]
+print(tp53_exons)
+```
+
+## Parse GFF with gffutils (Python)
+
+### Installation
+
+```bash
+pip install gffutils
+```
+
+### Create Database
+
+```python
+import gffutils
+
+# Create database (slow first time, fast for subsequent queries)
+db = gffutils.create_db('annotation.gff3', 'annotation.db',
+                         force=True, merge_strategy='create_unique')
+
+# Or load existing database
+db = gffutils.FeatureDB('annotation.db')
+```
+
+### Query Features
+
+```python
+import gffutils
+
+db = gffutils.FeatureDB('annotation.db')
+
+# Count features by type
+for featuretype in db.featuretypes():
+    count = db.count_features_of_type(featuretype)
+    print(f'{featuretype}: {count}')
+
+# Get all genes
+for gene in db.features_of_type('gene'):
+    print(f'{gene.id}: {gene.seqid}:{gene.start}-{gene.end}')
+
+# Get gene by ID
+gene = db['ENSG00000141510']  # TP53
+print(f'{gene.attributes["Name"][0]}: {gene.seqid}:{gene.start}-{gene.end}')
+
+# Get children (transcripts, exons)
+for transcript in db.children(gene, featuretype='mRNA'):
+    print(f'  Transcript: {transcript.id}')
+    for exon in db.children(transcript, featuretype='exon'):
+        print(f'    Exon: {exon.start}-{exon.end}')
+```
+
+### Get Introns
+
+```python
+import gffutils
+
+db = gffutils.FeatureDB('annotation.db')
+
+# Get introns for a transcript
+transcript = db['ENST00000269305']
+introns = list(db.interfeatures(db.children(transcript, featuretype='exon'),
+                                 new_featuretype='intron'))
+for intron in introns:
+    print(f'Intron: {intron.start}-{intron.end}')
+```
+
+## Convert Formats with gffread (CLI)
+
+### Installation
+
+```bash
+conda install -c bioconda gffread
+```
+
+### GTF to GFF3
+
+```bash
+gffread annotation.gtf -o annotation.gff3
+```
+
+### GFF3 to GTF
+
+```bash
+gffread annotation.gff3 -T -o annotation.gtf
+```
+
+### Extract Sequences
+
+```bash
+# Extract transcript sequences
+gffread -w transcripts.fa -g genome.fa annotation.gtf
+
+# Extract CDS sequences
+gffread -x cds.fa -g genome.fa annotation.gtf
+
+# Extract protein sequences
+gffread -y proteins.fa -g genome.fa annotation.gtf
+```
+
+### Filter Features
+
+```bash
+# Keep only protein-coding genes
+gffread annotation.gtf -C -o coding.gtf
+
+# Keep specific gene types
+gffread annotation.gtf --keep-genes=protein_coding -o coding.gtf
+```
+
+## Extract Regions with bedtools
+
+### Get Promoters
+
+```bash
+# Extract TSS (transcript start sites)
+awk '$3 == "transcript"' annotation.gtf | \
+    awk -v OFS='\t' '{
+        if ($7 == "+") print $1, $4-1, $4, ".", ".", $7;
+        else print $1, $5-1, $5, ".", ".", $7;
+    }' > tss.bed
+
+# Get promoter regions (2kb upstream of TSS)
+bedtools flank -i tss.bed -g genome.txt -l 2000 -r 0 -s > promoters.bed
+```
+
+### Get Gene Bodies
+
+```bash
+# Extract gene coordinates to BED
+awk '$3 == "gene"' annotation.gtf | \
+    awk -v OFS='\t' '{
+        split($0, a, "gene_id \""); split(a[2], b, "\"");
+        print $1, $4-1, $5, b[1], ".", $7;
+    }' > genes.bed
+```
+
+### Get Exons
+
+```bash
+# Extract unique exons
+awk '$3 == "exon"' annotation.gtf | \
+    awk -v OFS='\t' '{print $1, $4-1, $5, ".", ".", $7}' | \
+    sort -k1,1 -k2,2n | uniq > exons.bed
+```
+
+## Python: GTF to BED Conversion
+
+```python
+import gtfparse
+import pandas as pd
+
+def gtf_to_bed(gtf_path, feature_type='gene', output_path=None):
+    '''Convert GTF features to BED format.'''
+    df = gtfparse.read_gtf(gtf_path)
+    features = df[df['feature'] == feature_type].copy()
+
+    # Convert to 0-based coordinates
+    bed = pd.DataFrame({
+        'chrom': features['seqname'],
+        'start': features['start'] - 1,
+        'end': features['end'],
+        'name': features.get('gene_name', features.get('gene_id', '.')),
+        'score': 0,
+        'strand': features['strand']
+    })
+
+    if output_path:
+        bed.to_csv(output_path, sep='\t', header=False, index=False)
+    return bed
+
+# Usage
+genes_bed = gtf_to_bed('annotation.gtf', 'gene', 'genes.bed')
+exons_bed = gtf_to_bed('annotation.gtf', 'exon', 'exons.bed')
+```
+
+## Validate GTF/GFF
+
+```bash
+# Check GTF format
+gffread -E annotation.gtf
+
+# Check GFF3 format
+gffread -E annotation.gff3
+
+# Detailed validation
+gt gff3validator annotation.gff3  # requires genometools
+```
+
+## Common Attributes
+
+### GTF Attributes
+
+| Attribute | Description |
+|-----------|-------------|
+| gene_id | Ensembl gene ID |
+| gene_name | Gene symbol |
+| gene_biotype | protein_coding, lncRNA, etc. |
+| transcript_id | Ensembl transcript ID |
+| transcript_name | Transcript symbol |
+| exon_number | Exon position in transcript |
+| exon_id | Ensembl exon ID |
+
+### GFF3 Attributes
+
+| Attribute | Description |
+|-----------|-------------|
+| ID | Unique feature identifier |
+| Name | Display name |
+| Parent | Parent feature ID |
+| Dbxref | Database cross-references |
+| gene_biotype | Gene type |
+
+## Memory-Efficient Processing
+
+```python
+import gtfparse
+
+# Process large files in chunks (gtfparse loads all into memory)
+# For very large files, use gffutils database approach
+
+# Or filter during parsing
+df = gtfparse.read_gtf('annotation.gtf',
+                        features=['gene', 'exon'])  # Only load specific features
+```
+
+## Related Skills
+
+- bed-file-basics - BED format and conversion
+- interval-arithmetic - Gene/exon overlap analysis
+- proximity-operations - TSS proximity analysis
+- differential-expression/de-results - Gene coordinate mapping
+
+
+<!-- AUTHOR_SIGNATURE: 9a7f3c2e-MD-BABU-MIA-2026-MSSM-SECURE -->