@bgicli/bgicli 2.1.1 → 2.2.0

package/data/workflows/chip-atlas-peak-enrichment/SKILL.md

@@ -0,0 +1,212 @@
+---
+id: chip-atlas-peak-enrichment
+name: ChIP-Atlas Peak Enrichment
+category: epigenomics
+short-description: "Analyze enrichment of ChIP-seq peaks from 433,000+ experiments via the ChIP-Atlas API."
+detailed-description: "Analyze enrichment of ChIP-seq peaks from 433,000+ experiments via the official ChIP-Atlas Enrichment Analysis API. Submits gene lists for Fisher's exact test enrichment with Benjamini-Hochberg Q-values against all public ChIP-seq data. Generates 4-panel visualization. Supports 10 genomes - human (hg38, hg19), mouse (mm10, mm9), rat (rn6), fly (dm6, dm3), worm (ce11, ce10), yeast (sacCer3)."
+starting-prompt: Find ChIP-seq peak enrichment near my genes using ChIP-Atlas database . .
+---
+
+# ChIP-Atlas Peak Enrichment
+
+Find ChIP-seq peak enrichment near your genes using the official ChIP-Atlas Enrichment Analysis API.
+
+## When to Use This Skill
+
+Use ChIP-Atlas peak enrichment when you need to:
+- **Identify transcription factors binding near your genes** from DE analysis or pathway results
+- **Discover chromatin regulators** (TFs, histone modifications, chromatin remodelers) enriched near your gene set
+- **Validate regulatory relationships** between factors and target genes using public ChIP-seq data
+- **Find cell-type-specific regulators** by filtering to specific cell classes
+- **Query 433,000+ ChIP-seq experiments** via the official API without manual downloads
+
+**Don't use for:**
+- Direct ChIP-seq analysis from raw reads (use peak calling workflows)
+- Single gene lookups (use ChIP-Atlas web interface directly)
+- Offline analysis (requires internet for API calls)
+
+**Key Concept:** Submits your gene list to the ChIP-Atlas API, which performs Fisher's exact test enrichment analysis against all public ChIP-seq experiments. Returns fold enrichment, P-values, and BH-corrected Q-values.
+
+## Installation
+
+| Software | Version | License | Commercial Use | Installation |
+|----------|---------|---------|----------------|--------------|
+| pandas | >=1.3 | BSD-3-Clause | Permitted | `pip install pandas` |
+| requests | >=2.25 | Apache-2.0 | Permitted | `pip install requests` |
+| numpy | >=1.20 | BSD-3-Clause | Permitted | `pip install numpy` |
+| plotnine | >=0.10 | MIT | Permitted | `pip install plotnine` |
+| plotnine-prism | >=0.3 | MIT | Permitted | `pip install plotnine-prism` |
+
+```bash
+pip install pandas requests numpy plotnine plotnine-prism
+```
+
+**System requirements:** Internet connection (API calls to ChIP-Atlas and Ensembl)
+
+## Inputs
+
+**Gene list:**
+- Gene symbols (e.g., ["TP53", "MYC", "EGFR"])
+- Minimum: 3 genes; Recommended: 5-100 genes
+- Formats: Python list, plain text (one per line), CSV with gene column
+
+**Parameters:**
+- **Genome:** hg38 (default), hg19, mm10, mm9, rn6, dm6, dm3, ce11, ce10, sacCer3
+- **Antigen class:** "TFs and others" (default), "Histone", "ATAC-Seq", "DNase-seq", "RNA polymerase"
+- **Cell class:** "All cell types" (default), "Blood", "Neural", "Breast", etc.
+- **Threshold:** Peak-calling stringency (MACS2 -10×log10(p)): 50 (default, ~p<1e-5), 100 (~p<1e-10), 200 (~p<1e-20), 500 (~p<1e-50). Higher = fewer, more confident peaks. See [references/peak_thresholds.md](references/peak_thresholds.md).
+- **TSS window:** 5000bp upstream, 5000bp downstream (default)
+
+## Outputs
+
+**Analysis objects (Pickle):**
+- `analysis_object.pkl` - Complete results for downstream use
+  - Load with: `import pickle; obj = pickle.load(open('analysis_object.pkl', 'rb'))`
+  - Contains: enrichment_results, input_genes, input_regions, metadata, parameters
+
+**Results (CSV):**
+- `enrichment_results_all.csv` - All experiments (experiment_id, antigen, cell_type, fold_enrichment, p_value, q_value, overlap_rate, regions_with_overlaps, total_regions)
+- `enrichment_results_significant.csv` - Significant enrichments (q < 0.05, BH-corrected)
+- `enrichment_results_top20.csv` - Top 20 by significance (q-value, minimum 2 gene overlaps)
+
+**Visualizations (PNG + SVG):**
+- `chipatlas_enrichment.png/.svg` - 4-panel summary figure: top factors by significance, p-value distribution, overlap vs fold enrichment scatter, volcano plot (300 DPI)
+
+**Reports:**
+- `summary_report.md` - Human-readable analysis summary
+
+## Clarification Questions
+
+1. **Input Files** (ASK THIS FIRST):
+   - Do you have a gene list to analyze?
+   - Expected formats: Plain text (one gene per line), CSV with gene column, or DE results file
+   - **Or use example data?** `tp53_targets` (5 genes, fast test) or `immune_response` (20 genes)
+
+2. **Analysis parameters:**
+   - **Species/genome?** Human hg38 (default), hg19, mouse mm10/mm9, rat rn6, fly, worm, yeast
+   - **Experiment type?** "TFs and others" (default), "Histone", "ATAC-Seq"
+   - **Cell type class?** "All cell types" (default), or specific: "Blood", "Neural", "Breast", etc.
+   - **Peak threshold?** 50 (default, balanced), 100 (stringent), 200 (very stringent)
+   - **TSS window?** 5000bp up/down (default), or custom distance_up/distance_down
+
+## Standard Workflow
+
+**Step 1 - Load data:**
+```python
+# Option 1: Example data
+from scripts.load_example_data import load_example_data
+data = load_example_data("tp53_targets")
+gene_list = data['genes']
+
+# Option 2: Your own genes
+# from scripts.load_user_data import load_user_data
+# gene_list = load_user_data("my_genes.txt")
+```
+**VERIFICATION:** `"Data loaded successfully: {N} genes"`
+
+**Step 2 - Run enrichment analysis:**
+```python
+from scripts.run_enrichment_workflow import run_enrichment_workflow
+
+results = run_enrichment_workflow(
+    gene_list=gene_list,
+    genome="hg38",
+    antigen_class="TFs and others",
+    cell_class="All cell types",
+    threshold=50,
+    output_dir="chipatlas_results"
+)
+```
+**DO NOT write inline API query code. Just use the script.**
+
+**VERIFICATION:** `"Enrichment analysis completed successfully!"`
+
+**Step 3 - Generate visualizations:**
+```python
+from scripts.generate_all_plots import generate_all_plots
+generate_all_plots(results, output_dir="chipatlas_results", top_n=15)
+```
+**DO NOT write inline plotting code. The script handles PNG + SVG with graceful fallback.**
+
+**VERIFICATION:** `"All visualizations generated successfully!"`
+
+**Step 4 - Export results:**
+```python
+from scripts.export_all import export_all
+export_all(results, output_dir="chipatlas_results")
+```
+**DO NOT write custom export code. Use export_all().**
+
+**VERIFICATION:** `"=== Export Complete ==="`
+
+**IF SCRIPTS FAIL - Hierarchy:**
+1. **Fix and Retry (90%)** - Install missing package, check internet, re-run
+2. **Modify Script (5%)** - Edit the script, document changes
+3. **Use as Reference (4%)** - Read script, adapt approach
+4. **Write from Scratch (1%)** - Only if impossible, explain why
+
+## Common Issues
+
+| Error | Cause | Solution |
+|-------|-------|----------|
+| **API 400 error** | **Invalid parameters** | **Both antigenClass and cellClass must be non-empty. Use "All cell types" for all cells. See [references/chipatlas_metadata_format.md](references/chipatlas_metadata_format.md) for valid values.** |
+| **API timeout (>10 min)** | **Large analysis or server load** | **Normal for "All cell types" with many experiments. Wait up to 10 minutes. Reduce scope by selecting specific cell_class.** |
+| **Ensembl API timeout** | **Network or rate limiting** | **The workflow script automatically retries once after 60 seconds. If it still fails, the Ensembl step is skipped — enrichment results are unaffected but lack coordinate verification. Report this to the user (see interpretation rules).** |
+| **Gene not found** | **Retired symbol or typo** | **Script auto-retries with known aliases (e.g., IL8→CXCL8). Check HGNC (genenames.org) for current symbol. Only affects input_regions, not enrichment results.** |
+| **Fold enrichment >100,000x** | **Sentinel value (zero background overlap)** | **Not a real enrichment. Results are ranked by Q-value so these are deprioritized. Check overlap count — genuine hits have ≥2 gene overlaps.** |
+| **No significant enrichments** | **Gene set not enriched** | **Try: (1) Lower threshold (50), (2) Widen TSS window, (3) Check if genes are regulatory targets.** |
+| **SVG export error** | **Missing optional dependency** | **Normal - `generate_all_plots()` handles fallback automatically. Both PNG and SVG attempted; PNG always created.** |
+
+## Interpretation Guidelines
+
+**Q-value (BH-corrected, primary ranking):** <0.001 highly significant, <0.01 significant, <0.05 genome-wide significant
+**Fold Enrichment:** >10x very strong, 5-10x strong, 2-5x moderate, <2x weak, >100,000x likely sentinel value (zero background overlap)
+**Overlap Rate:** >50% core regulon, 20-50% key targets, <20% subset/indirect
+**Threshold:** Controls MACS2 peak-calling stringency. Default 50 ≈ p<1e-5. See [references/peak_thresholds.md](references/peak_thresholds.md).
+
+**⚠️ CRITICAL — When summarizing results to the user:**
+- **Explain threshold meaning on first mention.** When first discussing the threshold in your interpretation, state what it means in practical terms (e.g., "threshold=50 corresponds to MACS2 peak-calling at approximately p < 1e-5, balancing sensitivity and specificity"). Do not defer this explanation to a caveats section only.
+- **Cite overlap rates from `summary_report.md`.** Use the "Overlap Rate Summary" section for aggregate statistics. Do NOT round up or generalize (e.g., do not say "70–95%" if the report shows median 53%).
+- **Distinguish experiments from factors.** Multiple experiments for the same TF are independent datasets, not independent regulators. Use the "Top Factors (aggregated)" table when reporting unique factor counts.
+- **Acknowledge data availability bias.** If a factor has many experiments (>20), note that its prominence partly reflects being well-studied. Cite the experiment count from the aggregated table.
+- **Use Median FE (Sig), not Median FE (All), to judge enrichment strength.** The aggregated table has two FE columns: "Median FE (Sig)" uses only significant experiments (q < 0.05), while "Median FE (All)" averages across all experiments with ≥2 overlaps including non-significant ones. For factors with many experiments but few significant (e.g., Experiments=137, Sig=5), the "All" median is diluted by non-significant experiments and will dramatically understate actual enrichment. **Do not conclude "weak enrichment" from a low Median FE (All) when Median FE (Sig) shows strong enrichment.** Always cite Median FE (Sig) when discussing a factor's enrichment strength.
+- **Flag high experiment count with few significant.** If a factor has many experiments (>20) but few are significant (Sig column), note that most experiments for this factor do not show enrichment near these genes — only a subset of cell types/conditions do. Compare the Sig vs Experiments columns across factors to illustrate specificity differences.
+- **Report gene discrepancies.** If the summary report shows the API used fewer regions than genes submitted, mention this to the user with possible causes. Do NOT speculate about which specific gene was dropped — the API does not report this information.
+- **Report Ensembl vs API region count discrepancies prominently.** The Ensembl coordinate lookup and the ChIP-Atlas API use **different gene databases** (Ensembl vs RefSeq). Discrepancies are expected and should be explained clearly:
+  - If Ensembl mapped **0 genes** while the API analyzed N: explain that these are independent systems; enrichment results are valid but lack independent coordinate verification. Do not dismiss this as merely "optional."
+  - If Ensembl mapped **fewer genes** than the API (e.g., 4/5 vs 5/5): explain that the discrepancy reflects different database coverage — a gene may exist in RefSeq but have a failed/timed-out Ensembl lookup, or vice versa. State which system mapped how many genes and that the enrichment results use the API's own RefSeq mapping (not Ensembl).
+  - In both cases, the enrichment results themselves are valid. The Ensembl step provides independent verification only.
+- **Distinguish data-derived findings from background knowledge.** If citing known biology to interpret results, explicitly flag it as "from prior knowledge" or "based on known biology," not a conclusion from this analysis. Examples requiring explicit flagging: "TP63 and TP73 share the same DNA-binding domain as TP53" (protein family knowledge), "BRD4 is a bromodomain protein that binds acetylated histones at active promoters" (general factor biology), "these are canonical NF-κB targets" (pathway knowledge). Every such claim needs a phrase like "from prior knowledge of p53 family biology" — do not let background claims blend implicitly with data-derived findings.
+- **Report the total count of significant factors, not just the top 10.** The aggregated table shows the top 10, but the summary report header states the total count (e.g., "Top 10 of 29 significantly enriched factors"). Always report this total — do not say "10 factors were identified" when more exist. Mention notable omissions if biologically relevant factors appear in the full `enrichment_results_significant.csv`.
+- **Discuss all factors in the aggregated top table.** Every factor in the "Top Factors (aggregated)" table should be mentioned or briefly acknowledged. Do not silently omit factors — if one is less biologically interpretable, note that rather than skipping it.
+- **Note multiple testing across aggregated factors.** The Q-values in the aggregated table are BH-corrected across experiments, not across factors. Each factor's "best Q-value" is cherry-picked from its most significant experiment. Interpreting all 10 top factors as independent discoveries overstates confidence — note this when presenting the aggregated table (e.g., "these per-experiment Q-values do not account for testing across multiple factors").
+- **Small gene sets (≤10 genes): Lead with exploratory framing and connect every moderate-enrichment discussion to this caveat.** Frame the analysis as exploratory/demonstrative, not a powered study. **Your opening summary MUST lead with the exploratory nature** (e.g., "As an exploratory analysis with only 5 genes..." or "This demonstration-scale analysis with N genes identified...") rather than leading with the count of significant enrichments, which sounds more impressive than warranted at small N. Each gene contributes a large fraction of the overlap rate, so individual gene inclusion/exclusion substantially changes results. State this limitation prominently — do not bury it in a caveats section at the end. **Every time you discuss a factor with moderate fold enrichment (2–10x), you MUST explicitly note** that with only N input genes, a single gene's inclusion/exclusion could eliminate the signal entirely (e.g., "with only 5 input genes, this moderate enrichment should be interpreted with extra caution — removing a single gene could eliminate the signal").
+- **Cite ChIP-Atlas publications.** When presenting results, cite the database: Zou et al. (2024) for ChIP-Atlas 3.0 and Oki et al. (2018) for the original ChIP-Atlas. Include these in any written summary or report to acknowledge the data source.
+
+**Caveats (MUST include in any results summary):**
+- Results biased toward well-studied factors and common cell types. Heavily studied TFs may appear enriched partly due to data availability (more experiments = more chances to be significant).
+- Multiple experiments per factor are independent datasets, not independent biological signals. Use the aggregated factor table for deduplicated counts.
+- Results depend on the peak-calling threshold used. Discuss the threshold chosen and note that results may differ at other stringencies.
+- Validate key findings with orthogonal methods (expression, perturbation, motif analysis).
+
+## Suggested Next Steps
+
+1. **Threshold sensitivity check** — Re-run at threshold=100 or 200 to test whether top factors remain significant at more stringent peak-calling cutoffs. Offer this to the user as a robustness check (e.g., "Would you like me to re-run at threshold=100 to confirm these findings are robust?").
+2. **Validate top factors** with literature, expression correlation, perturbation data
+3. **Cell-type-specific analysis** with `cell_class` matching your experimental system
+4. **Motif analysis** of promoter regions for top factor binding motifs
+5. **Regulatory network** construction with top factors and target genes
+
+## Related Skills
+
+- **[gene-correlation-archs4](../gene-correlation-archs4/)** - Co-expression across 600K RNA-seq samples
+- **[grn-pyscenic](../grn-pyscenic/)** - Gene regulatory networks from single-cell data
+
+## References
+
+- Zou et al. (2024). ChIP-Atlas 3.0: a data-mining suite to explore chromosome architecture. *Nucleic Acids Research*. [doi:10.1093/nar/gkad884](https://doi.org/10.1093/nar/gkad884)
+- Zou et al. (2022). ChIP-Atlas 2021 update. *Nucleic Acids Research*. [doi:10.1093/nar/gkab933](https://doi.org/10.1093/nar/gkab933)
+- Oki et al. (2018). ChIP-Atlas: a data-mining suite. *EMBO Reports* 19(12):e46255. [doi:10.15252/embr.201846255](https://doi.org/10.15252/embr.201846255)
+- ChIP-Atlas: https://chip-atlas.org
+- API documentation: See [references/chipatlas_metadata_format.md](references/chipatlas_metadata_format.md)
+- Enrichment statistics: See [references/enrichment_statistics.md](references/enrichment_statistics.md)

package/data/workflows/chip-atlas-peak-enrichment/references/chipatlas_metadata_format.md

@@ -0,0 +1,115 @@
+# ChIP-Atlas API Format
+
+API endpoint, parameters, and response format for the ChIP-Atlas Enrichment
+Analysis.
+
+## API Endpoint
+
+**URL:** `POST https://dtn1.ddbj.nig.ac.jp/wabi/chipatlas/`
+
+**Model:** Asynchronous job (submit -> poll -> retrieve)
+
+**Website:** https://chip-atlas.org/enrichment_analysis
+
+## Submission Parameters
+
+| Parameter      | Type | Required | Description                                                       |
+| -------------- | ---- | -------- | ----------------------------------------------------------------- |
+| `address`      | str  | Yes      | Empty string `""`                                                 |
+| `format`       | str  | Yes      | `"json"` for JSON response                                        |
+| `result`       | str  | Yes      | `"www"`                                                           |
+| `genome`       | str  | Yes      | Genome assembly (see below)                                       |
+| `antigenClass` | str  | Yes      | Experiment type (must be non-empty)                               |
+| `cellClass`    | str  | Yes      | Cell type class (must be non-empty, use "All cell types" for all) |
+| `threshold`    | int  | Yes      | MACS2 threshold: 50, 100, 200, or 500                             |
+| `typeA`        | str  | Yes      | Input type: `"gene"` for gene list, `"bed"` for BED regions       |
+| `bedAFile`     | str  | Yes      | Newline-separated gene symbols or BED content                     |
+| `typeB`        | str  | Yes      | Comparison: `"refseq"` for RefSeq background                      |
+| `bedBFile`     | str  | Yes      | `"empty"` for refseq comparison                                   |
+| `permTime`     | int  | Yes      | Permutation count (1 for standard)                                |
+| `title`        | str  | Yes      | Job title (alphanumeric + underscore)                             |
+| `descriptionA` | str  | Yes      | Dataset A label                                                   |
+| `descriptionB` | str  | Yes      | Dataset B label                                                   |
+| `distanceUp`   | int  | Yes      | Bases upstream of TSS (default: 5000)                             |
+| `distanceDown` | int  | Yes      | Bases downstream of TSS (default: 5000)                           |
+
+## Supported Genomes
+
+| Genome     | Species                    |
+| ---------- | -------------------------- |
+| hg38, hg19 | _Homo sapiens_             |
+| mm10, mm9  | _Mus musculus_             |
+| rn6        | _Rattus norvegicus_        |
+| dm6, dm3   | _Drosophila melanogaster_  |
+| ce11, ce10 | _Caenorhabditis elegans_   |
+| sacCer3    | _Saccharomyces cerevisiae_ |
+
+## Valid Antigen Classes
+
+- `"TFs and others"` - Transcription factors and co-factors
+- `"Histone"` - Histone modifications (H3K4me3, H3K27ac, etc.)
+- `"RNA polymerase"` - RNA Pol II and variants
+- `"ATAC-Seq"` - Open chromatin
+- `"DNase-seq"` - DNase hypersensitivity
+- `"Bisulfite-Seq"` - DNA methylation
+- `"Input control"` - Control experiments
+- `"Annotation tracks"` - Genomic annotations
+
+## Valid Cell Type Classes (hg38)
+
+`"All cell types"`, `"Adipocyte"`, `"Blood"`, `"Bone"`, `"Breast"`,
+`"Cardiovascular"`, `"Digestive tract"`, `"Epidermis"`, `"Gonad"`, `"Kidney"`,
+`"Liver"`, `"Lung"`, `"Muscle"`, `"Neural"`, `"Others"`, `"Pancreas"`,
+`"Placenta"`, `"Pluripotent stem cell"`, `"Prostate"`, `"Uterus"`
+
+## Job Lifecycle
+
+### 1. Submit Job
+
+```python
+response = requests.post(WABI_API_URL, data={...})
+request_id = response.json()["requestId"]
+```
+
+### 2. Poll Status
+
+```
+GET {WABI_API_URL}{request_id}?info=status
+```
+
+Returns text with `status: running` or `status: finished`.
+
+### 3. Retrieve Results
+
+```
+GET {WABI_API_URL}{request_id}?info=result&format=tsv
+```
+
+## TSV Response Format
+
+11 tab-separated columns, no header:
+
+| Column | Name               | Type  | Example        |
+| ------ | ------------------ | ----- | -------------- |
+| 1      | Experiment ID      | str   | SRX21147142    |
+| 2      | Antigen class      | str   | TFs and others |
+| 3      | Antigen            | str   | TP53           |
+| 4      | Cell type class    | str   | Blood          |
+| 5      | Cell type          | str   | MOLM-13        |
+| 6      | Number of peaks    | int   | 99             |
+| 7      | Input overlap      | str   | 4/5            |
+| 8      | Background overlap | str   | 10/18851       |
+| 9      | log10(P-value)     | float | -12.0222       |
+| 10     | log10(Q-value)     | float | -8.15177       |
+| 11     | Fold enrichment    | float | 1508.08        |
+
+**Notes:**
+
+- P/Q values of 0 indicate p=1 (non-significant)
+- Fold enrichment of 1e-06 indicates depletion (0 overlap in query)
+- Results sorted by significance
+
+## References
+
+- Zou et al. (2024). "ChIP-Atlas 3.0." _Nucleic Acids Research_.
+- WABI API: https://github.com/inutano/chip-atlas/wiki

package/data/workflows/chip-atlas-peak-enrichment/references/enrichment_statistics.md

@@ -0,0 +1,145 @@
+# Enrichment Statistics
+
+Statistical methods used by the ChIP-Atlas Enrichment Analysis API.
+
+## Fisher's Exact Test
+
+The ChIP-Atlas API uses Fisher's exact test on a 2x2 contingency table to assess
+whether input genes are significantly enriched for binding by each ChIP-seq
+experiment compared to RefSeq background.
+
+### Contingency Table
+
+|                      | Overlapping peaks | No overlapping peaks |
+| -------------------- | :---------------: | :------------------: |
+| **Input genes**      |         a         |          b           |
+| **All RefSeq genes** |         c         |          d           |
+
+Where:
+
+- **a**: Number of input genes with promoter peaks (reported as numerator of
+  overlap_a)
+- **b**: Input genes without peaks
+- **c**: RefSeq genes with promoter peaks (from overlap_b)
+- **d**: RefSeq genes without peaks
+- Total RefSeq genes: ~18,622 (hg38)
+
+### P-value
+
+One-sided Fisher's exact test for enrichment (odds ratio > 1). API reports as
+log10(p-value), e.g., -12.02 means p = 10^-12.02.
+
+## Benjamini-Hochberg Q-values
+
+Multiple testing correction applied across all experiments tested:
+
+1. Rank all experiments by p-value
+2. Calculate adjusted p-value: `q_i = p_i * N / rank_i`
+3. Enforce monotonicity
+
+API reports as log10(q-value). Use q < 0.05 for genome-wide significance.
+
+## Fold Enrichment
+
+```
+fold_enrichment = (a / n_input) / (c / n_refseq)
+```
+
+Where n_input = total input genes, n_refseq = total RefSeq genes.
+
+### Interpretation
+
+| Fold Enrichment | Interpretation                                                |
+| --------------- | ------------------------------------------------------------- |
+| >10x            | Very strong enrichment, likely direct regulatory relationship |
+| 5-10x           | Strong enrichment, good evidence for regulation               |
+| 2-5x            | Moderate enrichment, possible regulation                      |
+| <2x             | Weak enrichment, may be background                            |
+| <1x             | Depletion (API reports as ~1e-06 for zero overlap)            |
+| >100,000x       | **Sentinel value** — see below                                |
+
+### Sentinel Values
+
+When the background overlap (c in the contingency table) is zero — meaning no
+RefSeq genes have peaks for that experiment in the queried regions — the fold
+enrichment formula produces division by zero. The API reports an extremely large
+value (typically 1,000,000x) as a sentinel.
+
+**How to identify sentinel values:**
+
+- Fold enrichment ≥ 100,000x
+- Typically low overlap count (e.g., 1/19 genes)
+- Often non-significant Q-value (q > 0.05)
+
+**How these are handled:**
+
+- Results are sorted by Q-value (not fold enrichment), so sentinels are
+  deprioritized
+- Top-20 and visualization panels require minimum 2 gene overlaps
+- Scatter and volcano plots cap fold enrichment at 1,000x for display
+
+## P-value Thresholds
+
+| P-value   | Significance Level          |
+| --------- | --------------------------- |
+| p < 0.001 | Highly significant (\*\*\*) |
+| p < 0.01  | Significant (\*\*)          |
+| p < 0.05  | Nominally significant (\*)  |
+| p >= 0.05 | Not significant (ns)        |
+
+## Overlap Rate
+
+Fraction of input genes with at least one overlapping peak from an experiment:
+
+```
+overlap_rate = regions_with_overlaps / total_input_genes
+```
+
+## API Output Columns
+
+The API returns 11 tab-separated columns (no header):
+
+1. Experiment ID (SRX...)
+2. Antigen class (e.g., "TFs and others")
+3. Antigen name
+4. Cell type class (e.g., "Blood")
+5. Cell type (e.g., "K-562")
+6. Number of peaks
+7. Input overlap (e.g., "4/5" = 4 of 5 input genes)
+8. Background overlap (e.g., "10/18851")
+9. log10(P-value)
+10. log10(Q-value)
+11. Fold enrichment
+
+## Data Biases
+
+ChIP-Atlas enrichment results are influenced by the composition of public
+ChIP-seq data:
+
+- **Well-studied factors are over-represented.** TFs like TP53, RELA (NF-κB),
+  and CTCF have hundreds to thousands of public experiments, while less-studied
+  factors may have only a few. A factor appearing enriched may partly reflect
+  data availability rather than biological specificity.
+- **Common cell types dominate.** Cell lines like K-562, HeLa, and MCF-7
+  contribute disproportionately to the database. Enrichment in these cell types
+  is more likely to be detected simply due to more available data.
+- **Publication bias.** Experiments in public databases skew toward positive
+  results and well-characterized regulatory relationships.
+
+**Recommendations:**
+
+- Use Q-value ranking (not fold enrichment) to prioritize results
+- Consider the number of available experiments when interpreting factor
+  importance
+- Validate key findings with orthogonal methods (expression data, perturbation
+  experiments, motif analysis)
+
+## References
+
+- Zou et al. (2024). "ChIP-Atlas 3.0: a data-mining suite to explore chromosome
+  architecture together with large-scale regulome data." _Nucleic Acids
+  Research_.
+- Zou et al. (2022). "ChIP-Atlas 2021 update: a data-mining suite for epigenomic
+  exploration." _Nucleic Acids Research_.
+- Oki et al. (2018). "ChIP-Atlas: a data-mining suite powered by full
+  integration of public ChIP-seq data." _EMBO Reports_.

package/data/workflows/chip-atlas-peak-enrichment/references/peak_thresholds.md

@@ -0,0 +1,63 @@
+# ChIP-Atlas Peak Calling Thresholds
+
+ChIP-Atlas provides pre-called peaks at four stringency levels based on MACS2
+significance scores.
+
+## Threshold Scale
+
+The API threshold parameter represents MACS2 `-10 * log10(p-value)`:
+
+| Threshold | MACS2 Score          | P-value (approx) | Stringency  | Recommended For                                     |
+| --------- | -------------------- | ---------------- | ----------- | --------------------------------------------------- |
+| **50**    | -10\*log10(p) >= 50  | 1e-5             | **Default** | Standard analysis, balanced sensitivity/specificity |
+| **100**   | -10\*log10(p) >= 100 | 1e-10            | High        | High-confidence peaks only                          |
+| **200**   | -10\*log10(p) >= 200 | 1e-20            | Very high   | Ultra-stringent, strongest binding only             |
+| **500**   | -10\*log10(p) >= 500 | 1e-50            | Extreme     | Exceptionally strong binding events                 |
+
+## Choosing a Threshold
+
+### Threshold 50 (1e-5) - **RECOMMENDED DEFAULT**
+
+- Standard enrichment analysis
+- Discovering new regulatory relationships
+- Works well for most transcription factors
+- Typical: ~5,000-10,000 peaks per TF experiment
+
+### Threshold 100 (1e-10) - High Stringency
+
+- High-confidence binding sites only
+- Minimizing false positives critical
+- Typical: ~1,000-5,000 peaks per experiment
+
+### Threshold 200 (1e-20) - Very High Stringency
+
+- Ultra-high-confidence peaks required
+- May miss weaker true positives
+- Typical: ~100-1,000 peaks per experiment
+
+### Threshold 500 (1e-50) - Extreme
+
+- Strongest binding events only
+- Very few peaks per experiment
+- Useful for histone marks with broad domains
+
+## Impact on Enrichment Analysis
+
+Higher thresholds produce fewer peaks but higher fold enrichments. Lower
+thresholds provide better sensitivity but include more noise. The API default of
+50 provides good balance.
+
+## Recommendations by Factor Type
+
+| Factor Type               | Recommended Threshold | Rationale                                  |
+| ------------------------- | --------------------- | ------------------------------------------ |
+| **Transcription Factors** | 50                    | Balance sensitivity/specificity            |
+| **Histone Modifications** | 50 or 100             | Broader regions, can use higher stringency |
+| **Chromatin Remodelers**  | 50                    | Often have weaker signals                  |
+| **ATAC-seq**              | 50                    | Standard open chromatin                    |
+
+## References
+
+- Zhang et al. (2008). "Model-based Analysis of ChIP-Seq (MACS)." _Genome
+  Biology_.
+- ChIP-Atlas documentation: https://chip-atlas.org

package/data/workflows/chip-atlas-peak-enrichment/references/promoter_definitions.md

@@ -0,0 +1,69 @@
+# Promoter Definitions and TSS Handling
+
+How genes are mapped to promoter regions for ChIP-seq peak overlap analysis.
+
+## API TSS Window (Default)
+
+The ChIP-Atlas API uses `distanceUp` and `distanceDown` parameters to define the
+TSS window:
+
+- **Default:** 5000bp upstream, 5000bp downstream (10kb total)
+- **API handles internally:** Gene-to-region conversion done server-side when
+  `typeA=gene`
+- **Ensembl lookup optional:** Only needed to populate `input_regions` for
+  downstream compatibility
+
+## Window Parameters
+
+| Window          | distanceUp | distanceDown | Total | Use Case                   |
+| --------------- | ---------- | ------------ | ----- | -------------------------- |
+| **API Default** | 5000bp     | 5000bp       | 10kb  | Broad regulatory scan      |
+| **Standard**    | 2000bp     | 500bp        | 2.5kb | Core promoter + proximal   |
+| **Narrow**      | 500bp      | 100bp        | 600bp | Core promoter only         |
+| **Wide**        | 10000bp    | 10000bp      | 20kb  | Distal regulatory elements |
+
+## Ensembl Gene Coordinate Lookup
+
+When `input_regions` are needed (for export_all.py compatibility), the batch
+Ensembl POST endpoint is used:
+
+```python
+# Batch lookup: single POST for all genes
+POST https://rest.ensembl.org/lookup/symbol/human
+Body: {"symbols": ["TP53", "CDKN1A", "BAX"]}
+```
+
+### TSS Calculation by Strand
+
+**Forward strand (+):** TSS = gene start coordinate **Reverse strand (-):** TSS
+= gene end coordinate
+
+```python
+if strand == 1:   # Forward
+    promoter = [TSS - distance_up, TSS + distance_down]
+else:              # Reverse
+    promoter = [TSS - distance_down, TSS + distance_up]
+```
+
+## Example: CDKN1A (p21)
+
+```
+Gene: CDKN1A
+Location: chr6:36,651,929-36,659,083 (GRCh38)
+Strand: + (forward)
+TSS: 36,651,929
+
+With API defaults (5000up/5000down):
+  Region: chr6:36,646,929-36,656,929
+```
+
+## Edge Cases
+
+- **Gene at chromosome start:** `promoter_start = max(1, tss - upstream)`
+- **Overlapping promoters:** Not merged; each gene independent
+- **Multiple transcripts:** Ensembl returns canonical transcript by default
+
+## References
+
+- Ensembl REST API: https://rest.ensembl.org
+- FANTOM5 CAGE promoter atlas

package/data/workflows/chip-atlas-peak-enrichment/scripts/__init__.py

@@ -0,0 +1 @@
+# chip-atlas-peak-enrichment scripts package