@synsci/cli-darwin-x64-baseline 1.1.76 → 1.1.78

package/bin/skills/ena-database/references/api_reference.md

@@ -0,0 +1,490 @@
+# ENA API Reference
+
+Comprehensive reference for the European Nucleotide Archive REST APIs.
+
+## ENA Portal API
+
+**Base URL:** `https://www.ebi.ac.uk/ena/portal/api`
+
+**Official Documentation:** https://www.ebi.ac.uk/ena/portal/api/doc
+
+### Search Endpoint
+
+**Endpoint:** `/search`
+
+**Method:** GET
+
+**Description:** Perform advanced searches across ENA data types with flexible filtering and formatting options.
+
+**Parameters:**
+
+| Parameter | Required | Description | Example |
+|-----------|----------|-------------|---------|
+| `result` | Yes | Data type to search | `sample`, `study`, `read_run`, `assembly`, `sequence`, `analysis`, `taxon` |
+| `query` | Yes | Search query using ENA query syntax | `tax_eq(9606)`, `study_accession="PRJNA123456"` |
+| `format` | No | Output format (default: tsv) | `json`, `tsv`, `xml` |
+| `fields` | No | Comma-separated list of fields to return | `accession,sample_title,scientific_name` |
+| `limit` | No | Maximum number of results (default: 100000) | `10`, `1000` |
+| `offset` | No | Result offset for pagination | `0`, `100` |
+| `sortFields` | No | Fields to sort by (comma-separated) | `accession`, `collection_date` |
+| `sortOrder` | No | Sort direction | `asc`, `desc` |
+| `dataPortal` | No | Restrict to specific data portal | `ena`, `pathogen`, `metagenome` |
+| `download` | No | Trigger file download | `true`, `false` |
+| `includeAccessions` | No | Comma-separated accessions to include | `SAMN01,SAMN02` |
+| `excludeAccessions` | No | Comma-separated accessions to exclude | `SAMN03,SAMN04` |
+
+**Query Syntax:**
+
+ENA uses a specialized query language with operators:
+
+- **Equality:** `field_name="value"` or `field_name=value`
+- **Wildcards:** `field_name="*partial*"` (use * for wildcard)
+- **Range:** `field_name>=value AND field_name<=value`
+- **Logical:** `query1 AND query2`, `query1 OR query2`, `NOT query`
+- **Taxonomy:** `tax_eq(taxon_id)` - exact match, `tax_tree(taxon_id)` - includes descendants
+- **Date ranges:** `collection_date>=2020-01-01 AND collection_date<=2023-12-31`
+- **In operator:** `study_accession IN (PRJNA1,PRJNA2,PRJNA3)`
+
+**Common Result Types:**
+
+- `study` - Research projects/studies
+- `sample` - Biological samples
+- `read_run` - Raw sequencing runs
+- `read_experiment` - Sequencing experiment metadata
+- `analysis` - Analysis results
+- `assembly` - Genome/transcriptome assemblies
+- `sequence` - Assembled sequences
+- `taxon` - Taxonomic records
+- `coding` - Protein coding sequences
+- `noncoding` - Non-coding sequences
+
+**Example Requests:**
+
+```python
+import requests
+
+# Search for human samples
+url = "https://www.ebi.ac.uk/ena/portal/api/search"
+params = {
+    "result": "sample",
+    "query": "tax_eq(9606)",
+    "format": "json",
+    "fields": "accession,sample_title,collection_date",
+    "limit": 100
+}
+response = requests.get(url, params=params)
+
+# Search for RNA-seq experiments in a study
+params = {
+    "result": "read_experiment",
+    "query": 'study_accession="PRJNA123456" AND library_strategy="RNA-Seq"',
+    "format": "tsv"
+}
+response = requests.get(url, params=params)
+
+# Find assemblies for E. coli with minimum contig N50
+params = {
+    "result": "assembly",
+    "query": "tax_tree(562) AND contig_n50>=50000",
+    "format": "json"
+}
+response = requests.get(url, params=params)
+```
+
+### Fields Endpoint
+
+**Endpoint:** `/returnFields`
+
+**Method:** GET
+
+**Description:** List available fields for a specific result type.
+
+**Parameters:**
+
+| Parameter | Required | Description | Example |
+|-----------|----------|-------------|---------|
+| `result` | Yes | Data type | `sample`, `study`, `assembly` |
+| `dataPortal` | No | Filter by data portal | `ena`, `pathogen` |
+
+**Example:**
+
+```python
+# Get all available fields for samples
+url = "https://www.ebi.ac.uk/ena/portal/api/returnFields"
+params = {"result": "sample"}
+response = requests.get(url, params=params)
+fields = response.json()
+```
+
+### Results Endpoint
+
+**Endpoint:** `/results`
+
+**Method:** GET
+
+**Description:** List available result types.
+
+**Example:**
+
+```python
+url = "https://www.ebi.ac.uk/ena/portal/api/results"
+response = requests.get(url)
+```
+
+### File Report Endpoint
+
+**Endpoint:** `/filereport`
+
+**Method:** GET
+
+**Description:** Get file information and download URLs for reads and analyses.
+
+**Parameters:**
+
+| Parameter | Required | Description | Example |
+|-----------|----------|-------------|---------|
+| `accession` | Yes | Run or analysis accession | `ERR123456` |
+| `result` | Yes | Must be `read_run` or `analysis` | `read_run` |
+| `format` | No | Output format | `json`, `tsv` |
+| `fields` | No | Fields to include | `run_accession,fastq_ftp,fastq_md5` |
+
+**Common File Report Fields:**
+
+- `run_accession` - Run accession number
+- `fastq_ftp` - FTP URLs for FASTQ files (semicolon-separated)
+- `fastq_aspera` - Aspera URLs for FASTQ files
+- `fastq_md5` - MD5 checksums (semicolon-separated)
+- `fastq_bytes` - File sizes in bytes (semicolon-separated)
+- `submitted_ftp` - FTP URLs for originally submitted files
+- `sra_ftp` - FTP URL for SRA format file
+
+**Example:**
+
+```python
+# Get FASTQ download URLs for a run
+url = "https://www.ebi.ac.uk/ena/portal/api/filereport"
+params = {
+    "accession": "ERR123456",
+    "result": "read_run",
+    "format": "json",
+    "fields": "run_accession,fastq_ftp,fastq_md5,fastq_bytes"
+}
+response = requests.get(url, params=params)
+file_info = response.json()
+
+# Download FASTQ files
+for ftp_url in file_info[0]['fastq_ftp'].split(';'):
+    # Download from ftp://ftp.sra.ebi.ac.uk/...
+    pass
+```
+
+## ENA Browser API
+
+**Base URL:** `https://www.ebi.ac.uk/ena/browser/api`
+
+**Official Documentation:** https://www.ebi.ac.uk/ena/browser/api/doc
+
+### XML Retrieval
+
+**Endpoint:** `/xml/{accession}`
+
+**Method:** GET
+
+**Description:** Retrieve record metadata in XML format.
+
+**Parameters:**
+
+| Parameter | Type | Description | Example |
+|-----------|------|-------------|---------|
+| `accession` | Path | Record accession number | `PRJNA123456`, `SAMEA123456`, `ERR123456` |
+| `download` | Query | Set to `true` to trigger download | `true` |
+| `includeLinks` | Query | Include cross-reference links | `true`, `false` |
+
+**Example:**
+
+```python
+# Get sample metadata in XML
+accession = "SAMEA123456"
+url = f"https://www.ebi.ac.uk/ena/browser/api/xml/{accession}"
+response = requests.get(url)
+xml_data = response.text
+
+# Get study with cross-references
+url = f"https://www.ebi.ac.uk/ena/browser/api/xml/PRJNA123456"
+params = {"includeLinks": "true"}
+response = requests.get(url, params=params)
+```
+
+### Text Retrieval
+
+**Endpoint:** `/text/{accession}`
+
+**Method:** GET
+
+**Description:** Retrieve sequences in EMBL flat file format.
+
+**Parameters:**
+
+| Parameter | Type | Description | Example |
+|-----------|------|-------------|---------|
+| `accession` | Path | Sequence accession | `LN847353` |
+| `download` | Query | Trigger download | `true` |
+| `expandDataclasses` | Query | Include related data classes | `true` |
+| `lineLimit` | Query | Limit output lines | `1000` |
+
+**Example:**
+
+```python
+# Get sequence in EMBL format
+url = "https://www.ebi.ac.uk/ena/browser/api/text/LN847353"
+response = requests.get(url)
+embl_format = response.text
+```
+
+### FASTA Retrieval
+
+**Endpoint:** `/fasta/{accession}`
+
+**Method:** GET
+
+**Description:** Retrieve sequences in FASTA format.
+
+**Parameters:**
+
+| Parameter | Type | Description | Example |
+|-----------|------|-------------|---------|
+| `accession` | Path | Sequence accession | `LN847353` |
+| `download` | Query | Trigger download | `true` |
+| `range` | Query | Subsequence range | `100-500` |
+| `lineLimit` | Query | Limit output lines | `1000` |
+
+**Example:**
+
+```python
+# Get full sequence
+url = "https://www.ebi.ac.uk/ena/browser/api/fasta/LN847353"
+response = requests.get(url)
+fasta_data = response.text
+
+# Get subsequence
+url = "https://www.ebi.ac.uk/ena/browser/api/fasta/LN847353"
+params = {"range": "1000-2000"}
+response = requests.get(url, params=params)
+```
+
+### Links Retrieval
+
+**Endpoint:** `/links/{source}/{accession}`
+
+**Method:** GET
+
+**Description:** Get cross-references to external databases.
+
+**Parameters:**
+
+| Parameter | Type | Description | Example |
+|-----------|------|-------------|---------|
+| `source` | Path | Source database type | `sample`, `study`, `sequence` |
+| `accession` | Path | Accession number | `SAMEA123456` |
+| `target` | Query | Target database filter | `sra`, `biosample` |
+
+**Example:**
+
+```python
+# Get all links for a sample
+url = "https://www.ebi.ac.uk/ena/browser/api/links/sample/SAMEA123456"
+response = requests.get(url)
+```
+
+## ENA Taxonomy REST API
+
+**Base URL:** `https://www.ebi.ac.uk/ena/taxonomy/rest`
+
+**Description:** Query taxonomic information including lineage and rank.
+
+### Tax ID Lookup
+
+**Endpoint:** `/tax-id/{taxon_id}`
+
+**Method:** GET
+
+**Description:** Get taxonomic information by NCBI taxonomy ID.
+
+**Example:**
+
+```python
+# Get E. coli taxonomy
+taxon_id = "562"
+url = f"https://www.ebi.ac.uk/ena/taxonomy/rest/tax-id/{taxon_id}"
+response = requests.get(url)
+taxonomy = response.json()
+# Returns: taxId, scientificName, commonName, rank, lineage, etc.
+```
+
+### Scientific Name Lookup
+
+**Endpoint:** `/scientific-name/{name}`
+
+**Method:** GET
+
+**Description:** Search by scientific name (may return multiple matches).
+
+**Example:**
+
+```python
+# Search by scientific name
+name = "Escherichia coli"
+url = f"https://www.ebi.ac.uk/ena/taxonomy/rest/scientific-name/{name}"
+response = requests.get(url)
+```
+
+### Suggest Names
+
+**Endpoint:** `/suggest-for-submission/{partial_name}`
+
+**Method:** GET
+
+**Description:** Get taxonomy suggestions for submission (autocomplete).
+
+**Example:**
+
+```python
+# Get suggestions
+partial = "Escheri"
+url = f"https://www.ebi.ac.uk/ena/taxonomy/rest/suggest-for-submission/{partial}"
+response = requests.get(url)
+```
+
+## Cross-Reference Service
+
+**Base URL:** `https://www.ebi.ac.uk/ena/xref/rest`
+
+**Description:** Access records related to ENA entries in external databases.
+
+### Get Cross-References
+
+**Endpoint:** `/json/{source}/{accession}`
+
+**Method:** GET
+
+**Description:** Retrieve cross-references in JSON format.
+
+**Parameters:**
+
+| Parameter | Type | Description | Example |
+|-----------|------|-------------|---------|
+| `source` | Path | Source database | `ena`, `sra` |
+| `accession` | Path | Accession number | `SRR000001` |
+
+**Example:**
+
+```python
+# Get cross-references for an SRA accession
+url = "https://www.ebi.ac.uk/ena/xref/rest/json/sra/SRR000001"
+response = requests.get(url)
+xrefs = response.json()
+```
+
+## CRAM Reference Registry
+
+**Base URL:** `https://www.ebi.ac.uk/ena/cram`
+
+**Description:** Retrieve reference sequences used in CRAM files.
+
+### MD5 Lookup
+
+**Endpoint:** `/md5/{md5_checksum}`
+
+**Method:** GET
+
+**Description:** Retrieve reference sequence by MD5 checksum.
+
+**Example:**
+
+```python
+# Get reference by MD5
+md5 = "7c3f69f0c5f0f0de6d7c34e7c2e25f5c"
+url = f"https://www.ebi.ac.uk/ena/cram/md5/{md5}"
+response = requests.get(url)
+reference_fasta = response.text
+```
+
+## Rate Limiting and Error Handling
+
+**Rate Limits:**
+- Maximum: 50 requests per second
+- Exceeding limit returns HTTP 429 (Too Many Requests)
+- Implement exponential backoff when receiving 429 responses
+
+**Common HTTP Status Codes:**
+
+- `200 OK` - Success
+- `204 No Content` - Success but no data returned
+- `400 Bad Request` - Invalid parameters
+- `404 Not Found` - Accession not found
+- `429 Too Many Requests` - Rate limit exceeded
+- `500 Internal Server Error` - Server error (retry with backoff)
+
+**Error Handling Pattern:**
+
+```python
+import time
+import requests
+from requests.adapters import HTTPAdapter
+from requests.packages.urllib3.util.retry import Retry
+
+def create_session_with_retries():
+    """Create requests session with retry logic"""
+    session = requests.Session()
+    retries = Retry(
+        total=5,
+        backoff_factor=1,
+        status_forcelist=[429, 500, 502, 503, 504],
+        allowed_methods=["GET", "POST"]
+    )
+    adapter = HTTPAdapter(max_retries=retries)
+    session.mount("https://", adapter)
+    return session
+
+# Usage
+session = create_session_with_retries()
+response = session.get(url, params=params)
+```
+
+## Bulk Download Recommendations
+
+For downloading large numbers of files or large datasets:
+
+1. **Use FTP directly** instead of API for file downloads
+   - Base FTP: `ftp://ftp.sra.ebi.ac.uk/vol1/fastq/`
+   - Aspera for high-speed: `era-fasp@fasp.sra.ebi.ac.uk:`
+
+2. **Use enaBrowserTools** command-line utility
+   ```bash
+   # Download by accession
+   enaDataGet ERR123456
+
+   # Download all runs from a study
+   enaGroupGet PRJEB1234
+   ```
+
+3. **Batch API requests** with proper delays
+   ```python
+   import time
+
+   accessions = ["ERR001", "ERR002", "ERR003"]
+   for acc in accessions:
+       response = requests.get(f"{base_url}/xml/{acc}")
+       # Process response
+       time.sleep(0.02)  # 50 req/sec = 0.02s between requests
+   ```
+
+## Query Optimization Tips
+
+1. **Use specific result types** instead of broad searches
+2. **Limit fields** to only what you need using `fields` parameter
+3. **Use pagination** for large result sets (limit + offset)
+4. **Cache taxonomy lookups** locally
+5. **Prefer JSON/TSV** over XML when possible (smaller, faster)
+6. **Use includeAccessions/excludeAccessions** to filter large result sets efficiently
+7. **Batch similar queries** together when possible