@bgicli/bgicli 2.1.1 → 2.2.0

package/data/workflows/chip-atlas-diff-analysis/SKILL.md

@@ -0,0 +1,222 @@
+---
+id: chip-atlas-diff-analysis
+name: ChIP-Atlas Diff Analysis
+category: epigenomics
+short-description: "Compare two groups of ChIP/ATAC/DNase-seq or Bisulfite-seq experiments to identify differential peak regions (DPR) or differentially methylated regions (DMR) via the ChIP-Atlas API."
+detailed-description: "Submit two groups of experiment accession IDs to ChIP-Atlas Diff Analysis API, which uses edgeR for differential peak regions (DPR) and metilene for differentially methylated regions (DMR). Returns BED files with genomic coordinates, logFC, p-values, and FDR-corrected q-values. Supports 10 genomes: human (hg38, hg19), mouse (mm10, mm9), rat (rn6), fly (dm6, dm3), worm (ce11, ce10), yeast (sacCer3)."
+starting-prompt: Compare two sets of ChIP-seq experiments to find differential peaks using ChIP-Atlas.
+---
+
+# ChIP-Atlas Diff Analysis
+
+Compare two groups of experiments to identify differential peak regions (DPR) or differentially methylated regions (DMR) using the ChIP-Atlas Diff Analysis API.
+
+## When to Use This Skill
+
+Use ChIP-Atlas diff analysis when you need to:
+- **Find differential peaks** between two conditions (treated vs control, tissue A vs B)
+- **Identify differentially methylated regions** between sample groups (Bisulfite-seq)
+- **Compare chromatin accessibility** between cell types using ATAC-seq/DNase-seq data
+- **Leverage edgeR-based statistical framework** on ChIP-Atlas public experiment data
+- **Avoid raw data downloads** — works directly with experiment accession IDs
+
+**Don't use for:**
+- Single experiment analysis or peak calling (use MACS2/MACS3 workflows)
+- Enrichment of factors near a gene list (use chip-atlas-peak-enrichment)
+- Offline analysis (requires internet for API calls)
+
+**Key Concept:** Submits two groups of experiment IDs to ChIP-Atlas. Server performs edgeR differential analysis (for DPR) or metilene (for DMR), returning BED files with genomic coordinates, logFC, p-values, q-values (FDR), and per-experiment normalized counts.
+
+## Installation
+
+| Software | Version | License | Commercial Use | Installation |
+|----------|---------|---------|----------------|--------------|
+| pandas | >=1.3 | BSD-3-Clause | Permitted | `pip install pandas` |
+| requests | >=2.25 | Apache-2.0 | Permitted | `pip install requests` |
+| numpy | >=1.20 | BSD-3-Clause | Permitted | `pip install numpy` |
+| plotnine | >=0.10 | MIT | Permitted | `pip install plotnine` |
+| plotnine-prism | >=0.2 | MIT | Permitted | `pip install plotnine-prism` |
+
+```bash
+pip install pandas requests numpy plotnine plotnine-prism
+```
+
+**System requirements:** Internet connection (API calls to ChIP-Atlas)
+
+## Inputs
+
+**Experiment IDs (two groups, minimum 2 per group):**
+- SRA accessions: SRX, ERX, DRX (e.g., SRX18419259)
+- GEO accessions: GSM (e.g., GSM6765200)
+- Formats: Python list, plain text (one per line), CSV with ID column
+
+**Parameters:**
+- **Genome:** hg38 (default), hg19, mm10, mm9, rn6, dm6, dm3, ce11, ce10, sacCer3
+- **Analysis type:** "diffbind" (default, DPR for ChIP/ATAC/DNase-seq) or "dmr" (DMR for Bisulfite-seq)
+
+## Outputs
+
+**Analysis objects (Pickle):**
+- `analysis_object.pkl` - Complete results for downstream use
+  - Load with: `import pickle; obj = pickle.load(open('analysis_object.pkl', 'rb'))`
+  - Contains: diff_regions, diff_regions_unfiltered, qc_warnings, experiments_a/b, raw_files, log_content, parameters
+
+**Results (CSV):**
+- `diff_regions_all.csv` - All differential regions, QC-filtered (chrom, start, end, logFC, pvalue, qvalue, direction, counts)
+- `diff_regions_significant.csv` - Significant regions (FDR < 0.05)
+- `diff_regions_top50.csv` - Top 50 by significance (with nearest gene annotation when available)
+- `diff_regions_unfiltered.csv` - Pre-QC regions (includes non-standard contigs and regions < 10bp)
+
+**Raw results:**
+- `raw_results/` - Original BED, IGV session, and log files from ChIP-Atlas
+
+**Visualizations (PNG + SVG, plotnine with Prism theme):**
+- `volcano_plot.png/.svg` - Volcano plot (logFC vs -log10 Q-value)
+- `chromosome_distribution.png/.svg` - Differential regions by chromosome (stacked bar)
+- `region_size_distribution.png/.svg` - Region size histogram (log-scale)
+- `ma_plot.png/.svg` - MA plot (mean counts vs logFC)
+
+**Reports:**
+- `summary_report.md` - Human-readable analysis summary with QC warnings, nearest gene annotations, and statistical caveats
+
+## Clarification Questions
+
+1. **Input Files** (ASK THIS FIRST):
+   - Do you have experiment IDs (SRX/GSM) for two groups to compare?
+   - Expected: At least 2 IDs per group (3+ recommended for statistical power)
+   - Formats: Plain text (one ID per line), CSV with ID column, or Python list
+   - **Or use example data?** `tp53_k562_dmso_vs_daunorubicin` (recommended: TP53 K-562 DMSO vs Daunorubicin, n=8 vs n=8) or `tp53_molm13_vs_k562` (MOLM-13 vs K-562, n=4 vs n=4, demonstrates sex-chr QC)
+
+2. **Analysis parameters:**
+   - **Species/genome?** Human hg38 (default), hg19, mouse mm10/mm9, rat rn6, fly, worm, yeast
+   - **Analysis type?** "diffbind" (default, for ChIP/ATAC/DNase-seq) or "dmr" (for Bisulfite-seq)
+   - **Group labels?** Names for each group (e.g., "Treated" vs "Control"). *(Auto-set for example data.)*
+
+## Standard Workflow
+
+🚨 **MANDATORY: USE SCRIPTS EXACTLY AS SHOWN - DO NOT WRITE INLINE CODE** 🚨
+
+**Step 1 - Load data:**
+```python
+# Option 1: Example data
+from scripts.load_example_data import load_example_data
+data = load_example_data("tp53_k562_dmso_vs_daunorubicin")
+experiments_a = data['experiments_a']
+experiments_b = data['experiments_b']
+
+# Option 2: Your own experiment IDs
+# from scripts.load_user_data import load_user_data
+# experiments_a = load_user_data("group_a.txt")
+# experiments_b = load_user_data("group_b.txt")
+```
+**VERIFICATION:** `"✓ Data loaded successfully"`
+
+**Step 2 - Run diff analysis:**
+```python
+from scripts.run_diff_workflow import run_diff_workflow
+
+results = run_diff_workflow(
+    experiments_a=experiments_a,
+    experiments_b=experiments_b,
+    genome=data.get('genome', 'hg38'),
+    analysis_type=data.get('analysis_type', 'diffbind'),
+    description_a=data.get('description_a', 'group_A'),
+    description_b=data.get('description_b', 'group_B'),
+    output_dir="diff_analysis_results"
+)
+```
+**DO NOT write inline API query code. Just use the script.**
+
+**VERIFICATION:** `"✓ Diff analysis completed successfully!"`
+
+**Step 3 - Generate visualizations:**
+```python
+from scripts.generate_all_plots import generate_all_plots
+generate_all_plots(results, output_dir="diff_analysis_results")
+```
+**DO NOT write inline plotting code. The script handles PNG + SVG with graceful fallback.**
+
+**VERIFICATION:** `"✓ All visualizations generated successfully!"`
+
+**Step 4 - Export results:**
+```python
+from scripts.export_all import export_all
+export_all(results, output_dir="diff_analysis_results")
+```
+**DO NOT write custom export code. Use export_all().**
+
+**VERIFICATION:** `"=== Export Complete ==="`
+
+⚠️ **IF SCRIPTS FAIL - Hierarchy:**
+1. **Fix and Retry (90%)** - Install missing package, check internet, re-run
+2. **Modify Script (5%)** - Edit the script, document changes
+3. **Use as Reference (4%)** - Read script, adapt approach
+4. **Write from Scratch (1%)** - Only if impossible, explain why
+
+## Common Issues
+
+| Error | Cause | Solution |
+|-------|-------|----------|
+| **API 400 error** | **Invalid parameters** | **Check experiment IDs are valid (SRX/ERX/DRX/GSM format). See [references/chipatlas_diff_api_format.md](references/chipatlas_diff_api_format.md).** |
+| **API timeout (>15 min)** | **Large dataset or server load** | **Normal for many experiments. Default timeout is 15 min. Reduce experiments per group if needed.** |
+| **No BED file in ZIP** | **Analysis produced no results** | **Check experiment IDs exist in ChIP-Atlas. Ensure both groups have valid data for the selected genome.** |
+| **Empty differential regions** | **No significant differences** | **Groups may be too similar. Try experiments from more distinct conditions.** |
+| **requestId parse error** | **API response format changed** | **Check ChIP-Atlas server status. The API uses text format (not JSON) for diff analysis.** |
+| **SVG export error** | **Missing optional dependency** | **Normal - `generate_all_plots()` handles fallback automatically. PNG always created.** |
+| **chrY/chrX regions all one-directional** | **Sex chromosome confound** | **Groups differ in sex (male vs female cell lines). QC check flags this automatically. Exclude sex chromosome regions from biological interpretation.** |
+| **Gene annotations missing** | **UCSC API unavailable** | **Normal in offline environments. Gene annotations are optional — all statistical results remain complete.** |
+
+## Interpretation Guidelines
+
+**logFC:** Log2 fold change — positive = enriched in Group A, negative = enriched in Group B
+**Q-value (FDR):** Benjamini-Hochberg corrected p-value — regions with FDR < 0.05 are significant
+**Counts:** TMM-normalized read counts per experiment (comma-separated in raw BED)
+**Region Size:** 100-500bp (TF sites), 500-2000bp (enhancers), >2000bp (broad marks)
+
+**Nearest gene annotations:** Top regions in the summary report and top50 CSV are annotated with overlapping or nearby genes via UCSC REST API. Shows "GENE" for direct overlap or "GENE (+2.1kb)" for nearest gene within 5kb. Annotation is optional and skipped gracefully if the UCSC API is unavailable.
+
+**Automated QC checks** (run automatically after Step 2):
+- **Sex chromosome confounds:** If chrY/chrX regions are overwhelmingly one-directional with near-zero counts in the depleted group, this signals a sex mismatch between groups (e.g., male vs female cell lines). Checks both all regions (>80% threshold) and significant regions specifically (>67% threshold) to catch cases where noisy non-significant regions mask the artifact. These peaks should NOT be interpreted as genuine differential binding.
+- **Non-standard contigs:** Regions on random scaffolds or unplaced sequences are filtered by default (mapping artifacts). Unfiltered data available in `diff_regions_unfiltered.csv`.
+- **Implausibly small regions:** Regions < 10bp are filtered by default (edge artifacts — TF motifs are ~20bp, histone marks 200bp+).
+- **Genomic clusters:** Dense clusters of one-directional significant regions (e.g., 10+ in <2 Mb) may indicate copy number amplification or structural variants rather than genuine differential binding. Investigate flagged clusters in a genome browser.
+- **chrM enrichment (ChIP-seq/ATAC-seq only):** Mitochondrial DNA lacks histones and canonical chromatin structure. Significant differential regions on chrM in ChIP-seq or ATAC-seq experiments almost always reflect mitochondrial DNA contamination or non-specific antibody binding, not genuine transcription factor binding. Regions on chrM that are also <20bp are doubly suspect (too small for a TF binding site). chrM regions are NOT flagged for Bisulfite-seq (DMR mode), where mitochondrial methylation differences can be biologically meaningful.
+- **Sample size:** With n ≤ 3, edgeR FDR values are unreliable — treat as approximate rankings. With n < 5, FDR estimates may be unstable. Focus on large effect sizes and validate key findings.
+- **MA plot asymmetry:** If significant hits in one direction cluster at substantially lower mean counts than the other direction (>1 log2 unit difference in median), this may indicate normalization artifacts or noise-driven signal. Focus biological interpretation on higher-count significant regions.
+- **Unusually small regions:** If the median significant region size is < 100bp for ChIP-seq/ATAC-seq, peak fragmentation may be inflating region counts. Check representative regions in a genome browser.
+
+**Reporting top hits:** When summarizing top differential regions, present them in **strict Q-value ranking order** without skipping unannotated regions. Regions lacking gene annotations are still statistically significant. If you curate a subset (e.g., only gene-annotated hits), explicitly label the table as "selected highlights" — never imply it is a strict significance ranking. **Accuracy:** When quoting specific statistics (logFC, Q-values, coordinates) in your summary, copy values directly from `summary_report.md` or the CSV files — do NOT re-derive, round, or paraphrase numerical values from memory, as this introduces transcription errors.
+
+**Gene annotations:** Nearest gene annotations indicate genomic proximity, not functional regulation. Describe associations factually (e.g., "a differential region near PCNA") without asserting direct regulatory relationships. Do not use individual gene annotations as validation of the analysis — gene proximity does not prove functional regulation.
+
+**Experimental design:** edgeR treats all experiments within a group as biological replicates. If experiments within a group differ in important ways (e.g., different genotypes, tissues, or treatments), state this clearly — the differential analysis captures the *average* effect between groups, and within-group heterogeneity is treated as noise. **When summarizing results to the user, design caveats MUST be stated prominently — not buried at the end. If within-group experiments are not true biological replicates, this is the single most important limitation to communicate.**
+
+**MA plot interpretation:** The MA plot shows mean expression (average log2 counts, x-axis) vs log fold change (y-axis). Normal pattern: funnel/fan shape narrowing at high counts (more power to detect small changes). Red flags: asymmetric significant hits clustering at low counts in one direction (potential normalization artifact), horizontal bands of significant hits (batch effects), or all significant hits at very low counts (noise-driven). **CRITICAL: Do NOT claim MA plot asymmetry in your summary unless the automated QC check flagged it.** The quantitative threshold is >1 log2 unit difference in median mean count between directions. Visual impressions of asymmetry below this threshold are not meaningful — small differences (e.g., 0.2 log2 units) are negligible and do not support claims of noise-driven signal. If no MA asymmetry QC warning appears, describe the MA plot as showing a normal pattern.
+
+**Biological interpretation of top hits:** When discussing top differential regions near known genes, briefly note the gene's known function for context but do NOT claim the differential peak proves direct regulation. Genomic proximity does not establish causation. For exploratory analyses, suggest formal pathway/GO enrichment analysis as a next step to identify systematic patterns. Distinguish between confirmatory hits (regions near genes expected to differ) and exploratory hits (unexpected associations requiring validation). When the target factor is well-characterized (e.g., TP53, CTCF, ESR1, NF-kB), note whether canonical target genes appear in the top differential regions — the **absence** of expected targets is itself a notable finding that may indicate the differential signal is dominated by artifacts or indirect effects. After excluding artifact regions, comment on the effect sizes (logFC) of remaining biologically interpretable hits — modest effect sizes (|logFC| < 1.5) combined with annotations near pseudogenes or lncRNAs suggest limited genuine differential binding.
+
+**Citations:** When writing your final summary, cite the data source (e.g., GSE accession) and key tools: ChIP-Atlas (Zou et al. 2024), edgeR (Robinson et al. 2010) for DPR, or metilene (Jühling et al. 2016) for DMR. Full references are in the References section below.
+
+**Caveats:** Results depend on available public data quality. Always check QC warnings in the summary report. Validate key regions with orthogonal methods. See [references/output_format.md](references/output_format.md) for detailed output format.
+
+## Suggested Next Steps
+
+1. **Visualize in IGV** using the `.igv.xml` session file or `.igv.bed` file
+2. **Annotate regions** with nearby genes using bedtools or GREAT
+3. **Motif analysis** on differential peaks (HOMER, MEME-ChIP)
+4. **Integrate with expression** data to link peaks to gene regulation
+
+## Related Skills
+
+- **[chip-atlas-peak-enrichment](../chip-atlas-peak-enrichment/)** - Enrichment of ChIP-seq peaks near gene lists
+- **[bulk-rnaseq-counts-to-de-deseq2](../bulk-rnaseq-counts-to-de-deseq2/)** - Differential expression to identify genes for downstream analysis
+
+## References
+
+- Zou et al. (2024). ChIP-Atlas 3.0: a data-mining suite to explore chromosome architecture. *Nucleic Acids Research*. [doi:10.1093/nar/gkad884](https://doi.org/10.1093/nar/gkad884)
+- Zou et al. (2022). ChIP-Atlas 2021 update. *Nucleic Acids Research*. [doi:10.1093/nar/gkab933](https://doi.org/10.1093/nar/gkab933)
+- Robinson et al. (2010). edgeR: a Bioconductor package for differential expression analysis. *Bioinformatics* 26(1):139-140.
+- Jühling et al. (2016). metilene: fast and sensitive calling of differentially methylated regions. *Genome Research* 26(2):256-262.
+- ChIP-Atlas: https://chip-atlas.org
+- API documentation: See [references/chipatlas_diff_api_format.md](references/chipatlas_diff_api_format.md)
+- Statistical methods: See [references/diff_analysis_methods.md](references/diff_analysis_methods.md)

package/data/workflows/chip-atlas-diff-analysis/references/chipatlas_diff_api_format.md

@@ -0,0 +1,106 @@
+# ChIP-Atlas Diff Analysis API Format
+
+## API Endpoint
+
+**Base URL:** `POST https://dtn1.ddbj.nig.ac.jp/wabi/chipatlas/`
+
+Same endpoint as enrichment analysis. The `antigenClass` parameter selects the
+analysis mode.
+
+## Required POST Parameters
+
+| Parameter      | Value                   | Description                                  |
+| -------------- | ----------------------- | -------------------------------------------- |
+| `address`      | `""`                    | Hidden but required (empty string)           |
+| `format`       | `"text"`                | Response format (NOT "json" like enrichment) |
+| `result`       | `"www"`                 | Hidden but required                          |
+| `genome`       | See below               | Target genome assembly                       |
+| `antigenClass` | `"diffbind"` or `"dmr"` | Analysis type                                |
+| `cellClass`    | `"empty"`               | Fixed value for diff analysis                |
+| `threshold`    | `"50"`                  | Fixed value                                  |
+| `typeA`        | `"srx"`                 | Input type for group A                       |
+| `bedAFile`     | Newline-separated IDs   | Experiment IDs for group A                   |
+| `typeB`        | `"srx"`                 | Input type for group B                       |
+| `bedBFile`     | Newline-separated IDs   | Experiment IDs for group B                   |
+| `permTime`     | `"1"`                   | Fixed value                                  |
+| `title`        | String                  | Analysis title                               |
+| `descriptionA` | String                  | Group A description                          |
+| `descriptionB` | String                  | Group B description                          |
+
+## Valid Genomes
+
+| Organism  | Assemblies |
+| --------- | ---------- |
+| Human     | hg38, hg19 |
+| Mouse     | mm10, mm9  |
+| Rat       | rn6        |
+| Fruit fly | dm6, dm3   |
+| Worm      | ce11, ce10 |
+| Yeast     | sacCer3    |
+
+## Analysis Types
+
+| antigenClass | Method   | Input Data                    |
+| ------------ | -------- | ----------------------------- |
+| `diffbind`   | edgeR    | ChIP-seq, ATAC-seq, DNase-seq |
+| `dmr`        | metilene | Bisulfite-seq                 |
+
+## Experiment ID Formats
+
+- **SRA:** SRX, ERX, DRX (e.g., `SRX18419259`)
+- **GEO:** GSM (e.g., `GSM6765200`)
+- Minimum 2 experiments per group
+
+## Response Format
+
+### Initial Submission (format=text)
+
+```
+requestId	wabi_chipatlas_2024-0124-1556-59-199-173766
+parameters	null
+current-time	2024-01-24 15:56:59
+start-time
+current-state
+```
+
+Parse `requestId` from the first line (tab-separated key-value).
+
+### Status Polling
+
+**URL:** `GET {BASE_URL}{requestId}?info=status`
+
+```
+status: running
+```
+
+or
+
+```
+status: finished
+```
+
+### Result Retrieval
+
+**URL:** `GET {BASE_URL}{requestId}?info=result&format=zip`
+
+Returns a ZIP archive. Files are inside a `wabi_result/` subdirectory:
+
+| File                  | Format      | Purpose                                                                    |
+| --------------------- | ----------- | -------------------------------------------------------------------------- |
+| `wabi_result.bed`     | 8-col TSV   | Main results: chrom, start, end, counts_A, counts_B, logFC, pvalue, qvalue |
+| `wabi_result.igv.bed` | BED9 + GFF3 | Visualization in IGV (color-coded by direction)                            |
+| `wabi_result.log`     | Text        | Analysis processing log (contains WABI_ID, sample info)                    |
+| `wabi_result.xml`     | XML         | IGV session file (loads BigWig tracks + regions)                           |
+| `README.md`           | Markdown    | Description of downloaded files                                            |
+
+**Note:** The main `.bed` file is NOT standard BED9 — it has 8 columns with
+statistical data. See [output_format.md](output_format.md) for detailed column
+definitions.
+
+## Key Differences from Enrichment API
+
+1. `format` is `"text"` not `"json"` — response must be parsed as text
+2. `cellClass` is always `"empty"` — no cell type filtering
+3. Both `bedAFile` and `bedBFile` contain experiment IDs (not gene lists)
+4. Results are ZIP (not TSV) — requires binary download and extraction
+5. Typical runtime is 2-10 minutes (longer than enrichment)

package/data/workflows/chip-atlas-diff-analysis/references/diff_analysis_methods.md

@@ -0,0 +1,89 @@
+# Diff Analysis Statistical Methods
+
+## Differential Peak Regions (DPR) — edgeR
+
+### Pipeline
+
+1. **Data retrieval:** BigWig and BED files retrieved from ChIP-Atlas based on
+   submitted experiment IDs
+2. **Format conversion:** BigWig files converted to bedGraph format
+3. **Coverage reconstruction:** Raw read coverage reconstructed using total
+   mapped read counts
+4. **Genome segmentation:** Genome segmented based on peak regions from query
+   datasets
+5. **Count matrix:** Read counts aggregated into m × n matrix (m regions × n
+   experiments)
+6. **Differential analysis:** edgeR package performs negative binomial GLM-based
+   testing
+7. **Result formatting:** Significant regions reported in BED9 format
+
+### Statistical Framework
+
+- **Model:** Negative binomial generalized linear model (edgeR)
+- **Normalization:** TMM (trimmed mean of M-values) across experiments
+- **Test:** Quasi-likelihood F-test for differential binding
+- **Multiple testing:** Benjamini-Hochberg FDR correction
+
+### Conceptual Relationship to DiffBind
+
+The approach is conceptually related to DiffBind but:
+
+- Does NOT require BAM files (uses pre-computed BigWig/BED from ChIP-Atlas)
+- Works directly with processed peak data
+- Enables comparison across any public experiments without raw data download
+
+### Reference
+
+Robinson MD, McCarthy DJ, Smyth GK. edgeR: a Bioconductor package for
+differential expression analysis of digital gene expression data.
+Bioinformatics. 2010;26(1):139-140.
+
+---
+
+## Differentially Methylated Regions (DMR) — metilene
+
+### Pipeline
+
+1. **Format conversion:** BigWig methylation data converted to bedGraph format
+2. **Data aggregation:** Methylation levels aggregated using `metilene_input.pl`
+3. **DMR detection:** DMRs detected using `metilene` with default parameters
+4. **Result formatting:** Significant DMRs reported in BED9 format
+
+### Statistical Framework
+
+- **Method:** Circular binary segmentation + Mann-Whitney U test
+- **Segmentation:** Identifies contiguous regions with differential methylation
+- **Significance:** p-values from two-sample tests at each CpG position
+- **Correction:** Multiple testing correction across detected regions
+
+### Reference
+
+Jühling F, et al. metilene: fast and sensitive calling of differentially
+methylated regions from bisulfite sequencing data. Genome Res.
+2016;26(2):256-262.
+
+---
+
+## Output Format
+
+### Main BED (wabi_result.bed) — 8-Column Custom Format
+
+The primary result is NOT standard BED9. It has 8 columns:
+
+```
+chrom  chromStart  chromEnd  counts_A  counts_B  logFC  pvalue  qvalue
+```
+
+- **counts_A/B:** Comma-separated TMM-normalized read counts per experiment
+- **logFC:** Log2 fold change from edgeR (positive = A enriched)
+- **pvalue:** edgeR p-value from quasi-likelihood F-test
+- **qvalue:** Benjamini-Hochberg FDR-corrected p-value
+
+### IGV BED (wabi_result.igv.bed) — BED9 + GFF3
+
+Standard BED9 format for IGV visualization:
+
+- **itemRgb:** Orange (222,131,68) = A-enriched, Blue (106,153,208) = B-enriched
+- **name field:** URL-encoded GFF3 attributes with LogFC, P-value, Q-value
+
+See [output_format.md](output_format.md) for complete column definitions.

package/data/workflows/chip-atlas-diff-analysis/references/output_format.md

@@ -0,0 +1,78 @@
+# ChIP-Atlas Diff Analysis Output Formats
+
+## Main Result BED (wabi_result.bed) — 8-Column Custom Format
+
+The primary result file is a custom 8-column tab-separated format (NOT standard
+BED9):
+
+| Column | Name       | Type  | Description                                                     |
+| ------ | ---------- | ----- | --------------------------------------------------------------- |
+| 1      | chrom      | str   | Chromosome (e.g., chr1, chrX)                                   |
+| 2      | chromStart | int   | Start position (0-based)                                        |
+| 3      | chromEnd   | int   | End position (exclusive)                                        |
+| 4      | counts_A   | str   | Comma-separated normalized counts for group A experiments       |
+| 5      | counts_B   | str   | Comma-separated normalized counts for group B experiments       |
+| 6      | logFC      | float | Log2 fold change (positive = A enriched, negative = B enriched) |
+| 7      | pvalue     | float | edgeR p-value                                                   |
+| 8      | qvalue     | float | Benjamini-Hochberg corrected q-value (FDR)                      |
+
+### Example Row
+
+```
+chr1	1048001	1048400	47.80,31.04,36.33	3.43,3.72,4.03	3.16	2.12e-11	7.39e-09
+```
+
+This means: Region chr1:1048001-1048400 has normalized counts ~38 in group A vs
+~3.7 in group B, logFC=3.16 (A enriched), highly significant (FDR=7.39e-09).
+
+### Derived Columns (computed by parse_bed_results.py)
+
+| Column       | Type  | Description                                            |
+| ------------ | ----- | ------------------------------------------------------ |
+| region_size  | int   | chromEnd - chromStart (bp)                             |
+| direction    | str   | 'A_enriched' (logFC > 0) or 'B_enriched' (logFC < 0)   |
+| significant  | bool  | qvalue < 0.05                                          |
+| mean_count_a | float | Mean of counts_A values                                |
+| mean_count_b | float | Mean of counts_B values                                |
+| score        | int   | -log10(qvalue) \* 100, capped at 1000 (BED-compatible) |
+
+## IGV Visualization BED (wabi_result.igv.bed) — BED9 + GFF3
+
+The IGV file uses standard BED9 format with URL-encoded GFF3 metadata:
+
+| Column | Name       | Type | Description                                           |
+| ------ | ---------- | ---- | ----------------------------------------------------- |
+| 1      | chrom      | str  | Chromosome                                            |
+| 2      | chromStart | int  | Start position                                        |
+| 3      | chromEnd   | int  | End position                                          |
+| 4      | name       | str  | URL-encoded GFF3 attributes (LogFC, P-value, Q-value) |
+| 5      | score      | int  | Significance score (0-1000)                           |
+| 6      | strand     | str  | Strand (always ".")                                   |
+| 7      | thickStart | int  | Display start (= chromStart)                          |
+| 8      | thickEnd   | int  | Display end (= chromEnd)                              |
+| 9      | itemRgb    | str  | RGB color indicating direction                        |
+
+### Direction from itemRgb Color
+
+| Color  | RGB         | Direction                            |
+| ------ | ----------- | ------------------------------------ |
+| Orange | 222,131,68  | Enriched in Group A (positive logFC) |
+| Blue   | 106,153,208 | Enriched in Group B (negative logFC) |
+
+## Region Size Interpretation
+
+| Size Range      | Typical Feature                           |
+| --------------- | ----------------------------------------- |
+| 100-500 bp      | Transcription factor binding sites        |
+| 500-2,000 bp    | Enhancers, narrow histone marks (H3K4me3) |
+| 2,000-10,000 bp | Broad histone marks (H3K27me3, H3K36me3)  |
+| >10,000 bp      | Super-enhancers, large chromatin domains  |
+
+## IGV Visualization
+
+1. Open IGV and load the matching genome (e.g., hg38)
+2. **Option A:** File → Open Session → select `wabi_result.xml` (loads
+   everything)
+3. **Option B:** File → Load from File → select `wabi_result.igv.bed` (regions
+   only)
+4. Regions appear colored orange (A-enriched) or blue (B-enriched)

package/data/workflows/chip-atlas-diff-analysis/scripts/__init__.py

@@ -0,0 +1 @@
+# chip-atlas-diff-analysis scripts package

package/data/workflows/chip-atlas-diff-analysis/scripts/annotate_genes.py

@@ -0,0 +1,144 @@
+"""
+Annotate differential regions with nearest/overlapping genes via UCSC REST API.
+
+Uses the UCSC Genome Browser REST API to find genes overlapping or near
+each genomic region. Requires internet (same as ChIP-Atlas API dependency).
+
+Graceful fallback: if UCSC API is unavailable, returns empty annotations
+without raising errors.
+"""
+
+import pandas as pd
+import requests
+
+UCSC_API_URL = "https://api.genome.ucsc.edu/getData/track"
+FLANK_BP = 5000
+
+
+def annotate_nearest_genes(df, genome="hg38", max_regions=50):
+    """
+    Annotate regions with overlapping/nearest gene symbols via UCSC API.
+
+    Parameters
+    ----------
+    df : pd.DataFrame
+        DataFrame with chrom, chromStart, chromEnd columns.
+        Should be pre-sorted by significance (top regions first).
+    genome : str
+        UCSC genome assembly (hg38, mm10, etc.)
+    max_regions : int
+        Maximum number of regions to annotate (default: 50).
+
+    Returns
+    -------
+    pd.Series
+        Gene annotations indexed to match df. Each entry is a string
+        like "SBF2" (overlapping) or "SBF2 (+2.1kb)" (nearest within flank).
+        Empty string for unannotated regions.
+    """
+    annotations = pd.Series("", index=df.index)
+
+    regions_to_query = df.head(max_regions)
+    if len(regions_to_query) == 0:
+        return annotations
+
+    print(f"   Annotating top {len(regions_to_query)} regions with nearest genes (UCSC API)...")
+
+    n_annotated = 0
+    n_failed = 0
+
+    for idx, row in regions_to_query.iterrows():
+        try:
+            gene_str = _query_genes_for_region(
+                row["chrom"],
+                int(row["chromStart"]),
+                int(row["chromEnd"]),
+                genome=genome,
+            )
+            if gene_str:
+                annotations.at[idx] = gene_str
+                n_annotated += 1
+        except Exception:
+            n_failed += 1
+            if n_failed >= 3:
+                print(f"   Warning: UCSC API unavailable ({n_failed} failures). "
+                      f"Skipping remaining gene annotations.")
+                break
+
+    print(f"   Annotated {n_annotated}/{len(regions_to_query)} regions with gene symbols")
+    return annotations
+
+
+def _query_genes_for_region(chrom, start, end, genome="hg38"):
+    """
+    Query UCSC REST API for genes overlapping or near a region.
+
+    Returns
+    -------
+    str
+        Gene symbol(s) or empty string.
+        Direct overlap: "GENE1, GENE2"
+        Nearest within flank: "GENE1 (+2.1kb)"
+    """
+    # First: check for direct overlap
+    genes = _fetch_gene_symbols(chrom, start, end, genome)
+    if genes:
+        return ", ".join(sorted(genes))
+
+    # Second: expand window and find nearest
+    expanded_start = max(0, start - FLANK_BP)
+    expanded_end = end + FLANK_BP
+    records = _fetch_gene_records(chrom, expanded_start, expanded_end, genome)
+
+    if not records:
+        return ""
+
+    # Find the closest gene by distance to region midpoint
+    region_mid = (start + end) / 2
+    best_gene = None
+    best_dist = float("inf")
+
+    for rec in records:
+        tx_start = rec.get("txStart", 0)
+        tx_end = rec.get("txEnd", 0)
+        gene_mid = (tx_start + tx_end) / 2
+        dist = abs(region_mid - gene_mid)
+        name2 = rec.get("name2", "")
+        if name2 and dist < best_dist:
+            best_dist = dist
+            best_gene = name2
+
+    if best_gene and best_dist > 0:
+        dist_kb = best_dist / 1000
+        return f"{best_gene} ({dist_kb:+.1f}kb)"
+    elif best_gene:
+        return best_gene
+
+    return ""
+
+
+def _fetch_gene_symbols(chrom, start, end, genome):
+    """Fetch unique gene symbols overlapping a region."""
+    records = _fetch_gene_records(chrom, start, end, genome)
+    symbols = set()
+    for rec in records:
+        name2 = rec.get("name2", "")
+        if name2:
+            symbols.add(name2)
+    return symbols
+
+
+def _fetch_gene_records(chrom, start, end, genome):
+    """Fetch raw gene records from UCSC REST API."""
+    url = (
+        f"{UCSC_API_URL}?genome={genome};track=ncbiRefSeq;"
+        f"chrom={chrom};start={int(start)};end={int(end)}"
+    )
+
+    try:
+        response = requests.get(url, timeout=10)
+        response.raise_for_status()
+        data = response.json()
+        return data.get("ncbiRefSeq", [])
+    except (requests.RequestException, ValueError, KeyError):
+        return []