@bgicli/bgicli 2.1.1 → 2.2.0

package/data/skills/instrument-data-to-allotrope/SKILL.md

@@ -0,0 +1,280 @@
+---
+name: instrument-data-to-allotrope
+description: Convert laboratory instrument output files (PDF, CSV, Excel, TXT) to Allotrope Simple Model (ASM) JSON format or flattened 2D CSV. Use this skill when scientists need to standardize instrument data for LIMS systems, data lakes, or downstream analysis. Supports auto-detection of instrument types. Outputs include full ASM JSON, flattened CSV for easy import, and exportable Python code for data engineers. Common triggers include converting instrument files, standardizing lab data, preparing data for upload to LIMS/ELN systems, or generating parser code for production pipelines.
+---
+
+# Instrument Data to Allotrope Converter
+
+Convert instrument files into standardized Allotrope Simple Model (ASM) format for LIMS upload, data lakes, or handoff to data engineering teams.
+
+> **Note: This is an Example Skill**
+>
+> This skill demonstrates how skills can support your data engineering tasks—automating schema transformations, parsing instrument outputs, and generating production-ready code.
+>
+> **To customize for your organization:**
+> - Modify the `references/` files to include your company's specific schemas or ontology mappings
+> - Use an MCP server to connect to systems that define your schemas (e.g., your LIMS, data catalog, or schema registry)
+> - Extend the `scripts/` to handle proprietary instrument formats or internal data standards
+>
+> This pattern can be adapted for any data transformation workflow where you need to convert between formats or validate against organizational standards.
+
+## Workflow Overview
+
+1. **Detect instrument type** from file contents (auto-detect or user-specified)
+2. **Parse file** using allotropy library (native) or flexible fallback parser
+3. **Generate outputs**:
+   - ASM JSON (full semantic structure)
+   - Flattened CSV (2D tabular format)
+   - Python parser code (for data engineer handoff)
+4. **Deliver** files with summary and usage instructions
+
+> **When Uncertain:** If you're unsure how to map a field to ASM (e.g., is this raw data or calculated? device setting or environmental condition?), ask the user for clarification. Refer to `references/field_classification_guide.md` for guidance, but when ambiguity remains, confirm with the user rather than guessing.
+
+## Quick Start
+
+```python
+# Install requirements first
+pip install allotropy pandas openpyxl pdfplumber --break-system-packages
+
+# Core conversion
+from allotropy.parser_factory import Vendor
+from allotropy.to_allotrope import allotrope_from_file
+
+# Convert with allotropy
+asm = allotrope_from_file("instrument_data.csv", Vendor.BECKMAN_VI_CELL_BLU)
+```
+
+## Output Format Selection
+
+**ASM JSON (default)** - Full semantic structure with ontology URIs
+- Best for: LIMS systems expecting ASM, data lakes, long-term archival
+- Validates against Allotrope schemas
+
+**Flattened CSV** - 2D tabular representation
+- Best for: Quick analysis, Excel users, systems without JSON support
+- Each measurement becomes one row with metadata repeated
+
+**Both** - Generate both formats for maximum flexibility
+
+## Calculated Data Handling
+
+**IMPORTANT:** Separate raw measurements from calculated/derived values.
+
+- **Raw data** → `measurement-document` (direct instrument readings)
+- **Calculated data** → `calculated-data-aggregate-document` (derived values)
+
+Calculated values MUST include traceability via `data-source-aggregate-document`:
+
+```json
+"calculated-data-aggregate-document": {
+  "calculated-data-document": [{
+    "calculated-data-identifier": "SAMPLE_B1_DIN_001",
+    "calculated-data-name": "DNA integrity number",
+    "calculated-result": {"value": 9.5, "unit": "(unitless)"},
+    "data-source-aggregate-document": {
+      "data-source-document": [{
+        "data-source-identifier": "SAMPLE_B1_MEASUREMENT",
+        "data-source-feature": "electrophoresis trace"
+      }]
+    }
+  }]
+}
+```
+
+**Common calculated fields by instrument type:**
+| Instrument | Calculated Fields |
+|------------|-------------------|
+| Cell counter | Viability %, cell density dilution-adjusted values |
+| Spectrophotometer | Concentration (from absorbance), 260/280 ratio |
+| Plate reader | Concentrations from standard curve, %CV |
+| Electrophoresis | DIN/RIN, region concentrations, average sizes |
+| qPCR | Relative quantities, fold change |
+
+See `references/field_classification_guide.md` for detailed guidance on raw vs. calculated classification.
+
+## Validation
+
+Always validate ASM output before delivering to the user:
+
+```bash
+python scripts/validate_asm.py output.json
+python scripts/validate_asm.py output.json --reference known_good.json  # Compare to reference
+python scripts/validate_asm.py output.json --strict  # Treat warnings as errors
+```
+
+**Validation Rules:**
+- Based on Allotrope ASM specification (December 2024)
+- Last updated: 2026-01-07
+- Source: https://gitlab.com/allotrope-public/asm
+
+**Soft Validation Approach:**
+Unknown techniques, units, or sample roles generate **warnings** (not errors) to allow for forward compatibility. If Allotrope adds new values after December 2024, the validator won't block them—it will flag them for manual verification. Use `--strict` mode to treat warnings as errors if you need stricter validation.
+
+**What it checks:**
+- Correct technique selection (e.g., multi-analyte profiling vs plate reader)
+- Field naming conventions (space-separated, not hyphenated)
+- Calculated data has traceability (`data-source-aggregate-document`)
+- Unique identifiers exist for measurements and calculated values
+- Required metadata present
+- Valid units and sample roles (with soft validation for unknown values)
+
+## Supported Instruments
+
+See `references/supported_instruments.md` for complete list. Key instruments:
+
+| Category | Instruments |
+|----------|-------------|
+| Cell Counting | Vi-CELL BLU, Vi-CELL XR, NucleoCounter |
+| Spectrophotometry | NanoDrop One/Eight/8000, Lunatic |
+| Plate Readers | SoftMax Pro, EnVision, Gen5, CLARIOstar |
+| ELISA | SoftMax Pro, BMG MARS, MSD Workbench |
+| qPCR | QuantStudio, Bio-Rad CFX |
+| Chromatography | Empower, Chromeleon |
+
+## Detection & Parsing Strategy
+
+### Tier 1: Native allotropy parsing (PREFERRED)
+**Always try allotropy first.** Check available vendors directly:
+
+```python
+from allotropy.parser_factory import Vendor
+
+# List all supported vendors
+for v in Vendor:
+    print(f"{v.name}")
+
+# Common vendors:
+# AGILENT_TAPESTATION_ANALYSIS  (for TapeStation XML)
+# BECKMAN_VI_CELL_BLU
+# THERMO_FISHER_NANODROP_EIGHT
+# MOLDEV_SOFTMAX_PRO
+# APPBIO_QUANTSTUDIO
+# ... many more
+```
+
+**When the user provides a file, check if allotropy supports it before falling back to manual parsing.** The `scripts/convert_to_asm.py` auto-detection only covers a subset of allotropy vendors.
+
+### Tier 2: Flexible fallback parsing
+**Only use if allotropy doesn't support the instrument.** This fallback:
+- Does NOT generate `calculated-data-aggregate-document`
+- Does NOT include full traceability
+- Produces simplified ASM structure
+
+Use flexible parser with:
+- Column name fuzzy matching
+- Unit extraction from headers
+- Metadata extraction from file structure
+
+### Tier 3: PDF extraction
+For PDF-only files, extract tables using pdfplumber, then apply Tier 2 parsing.
+
+## Pre-Parsing Checklist
+
+Before writing a custom parser, ALWAYS:
+
+1. **Check if allotropy supports it** - Use native parser if available
+2. **Find a reference ASM file** - Check `references/examples/` or ask user
+3. **Review instrument-specific guide** - Check `references/instrument_guides/`
+4. **Validate against reference** - Run `validate_asm.py --reference <file>`
+
+## Common Mistakes to Avoid
+
+| Mistake | Correct Approach |
+|---------|------------------|
+| Manifest as object | Use URL string |
+| Lowercase detection types | Use "Absorbance" not "absorbance" |
+| "emission wavelength setting" | Use "detector wavelength setting" for emission |
+| All measurements in one document | Group by well/sample location |
+| Missing procedure metadata | Extract ALL device settings per measurement |
+
+## Code Export for Data Engineers
+
+Generate standalone Python scripts that scientists can hand off:
+
+```python
+# Export parser code
+python scripts/export_parser.py --input "data.csv" --vendor "VI_CELL_BLU" --output "parser_script.py"
+```
+
+The exported script:
+- Has no external dependencies beyond pandas/allotropy
+- Includes inline documentation
+- Can run in Jupyter notebooks
+- Is production-ready for data pipelines
+
+## File Structure
+
+```
+instrument-data-to-allotrope/
+├── SKILL.md                          # This file
+├── scripts/
+│   ├── convert_to_asm.py            # Main conversion script
+│   ├── flatten_asm.py               # ASM → 2D CSV conversion
+│   ├── export_parser.py             # Generate standalone parser code
+│   └── validate_asm.py              # Validate ASM output quality
+└── references/
+    ├── supported_instruments.md     # Full instrument list with Vendor enums
+    ├── asm_schema_overview.md       # ASM structure reference
+    ├── field_classification_guide.md # Where to put different field types
+    └── flattening_guide.md          # How flattening works
+```
+
+## Usage Examples
+
+### Example 1: Vi-CELL BLU file
+```
+User: "Convert this cell counting data to Allotrope format"
+[uploads viCell_Results.xlsx]
+
+Claude:
+1. Detects Vi-CELL BLU (95% confidence)
+2. Converts using allotropy native parser
+3. Outputs:
+   - viCell_Results_asm.json (full ASM)
+   - viCell_Results_flat.csv (2D format)
+   - viCell_parser.py (exportable code)
+```
+
+### Example 2: Request for code handoff
+```
+User: "I need to give our data engineer code to parse NanoDrop files"
+
+Claude:
+1. Generates self-contained Python script
+2. Includes sample input/output
+3. Documents all assumptions
+4. Provides Jupyter notebook version
+```
+
+### Example 3: LIMS-ready flattened output
+```
+User: "Convert this ELISA data to a CSV I can upload to our LIMS"
+
+Claude:
+1. Parses plate reader data
+2. Generates flattened CSV with columns:
+   - sample_identifier, well_position, measurement_value, measurement_unit
+   - instrument_serial_number, analysis_datetime, assay_type
+3. Validates against common LIMS import requirements
+```
+
+## Implementation Notes
+
+### Installing allotropy
+```bash
+pip install allotropy --break-system-packages
+```
+
+### Handling parse failures
+If allotropy native parsing fails:
+1. Log the error for debugging
+2. Fall back to flexible parser
+3. Report reduced metadata completeness to user
+4. Suggest exporting different format from instrument
+
+### ASM Schema Validation
+Validate output against Allotrope schemas when available:
+```python
+import jsonschema
+# Schema URLs in references/asm_schema_overview.md
+```

package/data/skills/interpro-database/SKILL.md

@@ -0,0 +1,305 @@
+---
+name: interpro-database
+description: Query InterPro for protein family, domain, and functional site annotations. Integrates Pfam, PANTHER, PRINTS, SMART, SUPERFAMILY, and 11 other member databases. Use for protein function prediction, domain architecture analysis, evolutionary classification, and GO term mapping.
+license: CC0-1.0
+metadata:
+    skill-author: Kuan-lin Huang
+---
+
+# InterPro Database
+
+## Overview
+
+InterPro (https://www.ebi.ac.uk/interpro/) is a comprehensive resource for protein family and domain classification maintained by EMBL-EBI. It integrates signatures from 13 member databases including Pfam, PANTHER, PRINTS, ProSite, SMART, TIGRFAM, SUPERFAMILY, CDD, and others, providing a unified view of protein functional annotations for over 100 million protein sequences.
+
+InterPro classifies proteins into:
+- **Families**: Groups of proteins sharing common ancestry and function
+- **Domains**: Independently folding structural/functional units
+- **Homologous superfamilies**: Structurally similar protein regions
+- **Repeats**: Short tandem sequences
+- **Sites**: Functional sites (active, binding, PTM)
+
+**Key resources:**
+- InterPro website: https://www.ebi.ac.uk/interpro/
+- REST API: https://www.ebi.ac.uk/interpro/api/
+- API documentation: https://github.com/ProteinsWebTeam/interpro7-api/blob/master/docs/
+- Python client: via `requests`
+
+## When to Use This Skill
+
+Use InterPro when:
+
+- **Protein function prediction**: What function(s) does an uncharacterized protein likely have?
+- **Domain architecture**: What domains make up a protein, and in what order?
+- **Protein family classification**: Which family/superfamily does a protein belong to?
+- **GO term annotation**: Map protein sequences to Gene Ontology terms via InterPro
+- **Evolutionary analysis**: Are two proteins in the same homologous superfamily?
+- **Structure prediction context**: What domains should a new protein structure be compared against?
+- **Pipeline annotation**: Batch-annotate proteomes or novel sequences
+
+## Core Capabilities
+
+### 1. InterPro REST API
+
+Base URL: `https://www.ebi.ac.uk/interpro/api/`
+
+```python
+import requests
+
+BASE_URL = "https://www.ebi.ac.uk/interpro/api"
+
+def interpro_get(endpoint, params=None):
+    url = f"{BASE_URL}/{endpoint}"
+    headers = {"Accept": "application/json"}
+    response = requests.get(url, params=params, headers=headers)
+    response.raise_for_status()
+    return response.json()
+```
+
+### 2. Look Up a Protein
+
+```python
+def get_protein_entries(uniprot_id):
+    """Get all InterPro entries that match a UniProt protein."""
+    data = interpro_get(f"protein/UniProt/{uniprot_id}/entry/InterPro/")
+    return data
+
+# Example: Human p53 (TP53)
+result = get_protein_entries("P04637")
+entries = result.get("results", [])
+
+for entry in entries:
+    meta = entry["metadata"]
+    print(f"  {meta['accession']} ({meta['type']}): {meta['name']}")
+    # e.g., IPR011615 (domain): p53, tetramerisation domain
+    #       IPR010991 (domain): p53, DNA-binding domain
+    #       IPR013872 (family): p53 family
+```
+
+### 3. Get Specific InterPro Entry
+
+```python
+def get_entry(interpro_id):
+    """Fetch details for an InterPro entry."""
+    return interpro_get(f"entry/InterPro/{interpro_id}/")
+
+# Example: Get Pfam domain PF00397 (WW domain)
+ww_entry = get_entry("IPR001202")
+print(f"Name: {ww_entry['metadata']['name']}")
+print(f"Type: {ww_entry['metadata']['type']}")
+
+# Also supports member database IDs:
+def get_pfam_entry(pfam_id):
+    return interpro_get(f"entry/Pfam/{pfam_id}/")
+
+pfam = get_pfam_entry("PF00397")
+```
+
+### 4. Search Proteins by InterPro Entry
+
+```python
+def get_proteins_for_entry(interpro_id, database="UniProt", page_size=25):
+    """Get all proteins annotated with an InterPro entry."""
+    params = {"page_size": page_size}
+    data = interpro_get(f"entry/InterPro/{interpro_id}/protein/{database}/", params)
+    return data
+
+# Example: Find all human kinase-domain proteins
+kinase_proteins = get_proteins_for_entry("IPR000719")  # Protein kinase domain
+print(f"Total proteins: {kinase_proteins['count']}")
+```
+
+### 5. Domain Architecture
+
+```python
+def get_domain_architecture(uniprot_id):
+    """Get the complete domain architecture of a protein."""
+    data = interpro_get(f"protein/UniProt/{uniprot_id}/")
+    return data
+
+# Example: Get full domain architecture for EGFR
+egfr = get_domain_architecture("P00533")
+
+# The response includes locations of all matching entries on the sequence
+for entry in egfr.get("entries", []):
+    for fragment in entry.get("entry_protein_locations", []):
+        for loc in fragment.get("fragments", []):
+            print(f"  {entry['accession']}: {loc['start']}-{loc['end']}")
+```
+
+### 6. GO Term Mapping
+
+```python
+def get_go_terms_for_protein(uniprot_id):
+    """Get GO terms associated with a protein via InterPro."""
+    data = interpro_get(f"protein/UniProt/{uniprot_id}/")
+
+    # GO terms are embedded in the entry metadata
+    go_terms = []
+    for entry in data.get("entries", []):
+        go = entry.get("metadata", {}).get("go_terms", [])
+        go_terms.extend(go)
+
+    # Deduplicate
+    seen = set()
+    unique_go = []
+    for term in go_terms:
+        if term["identifier"] not in seen:
+            seen.add(term["identifier"])
+            unique_go.append(term)
+
+    return unique_go
+
+# GO terms include:
+# {"identifier": "GO:0004672", "name": "protein kinase activity", "category": {"code": "F", "name": "Molecular Function"}}
+```
+
+### 7. Batch Protein Lookup
+
+```python
+def batch_lookup_proteins(uniprot_ids, database="UniProt"):
+    """Look up multiple proteins and collect their InterPro entries."""
+    import time
+    results = {}
+    for uid in uniprot_ids:
+        try:
+            data = interpro_get(f"protein/{database}/{uid}/entry/InterPro/")
+            entries = data.get("results", [])
+            results[uid] = [
+                {
+                    "accession": e["metadata"]["accession"],
+                    "name": e["metadata"]["name"],
+                    "type": e["metadata"]["type"]
+                }
+                for e in entries
+            ]
+        except Exception as e:
+            results[uid] = {"error": str(e)}
+        time.sleep(0.3)  # Rate limiting
+    return results
+
+# Example
+proteins = ["P04637", "P00533", "P38398", "Q9Y6I9"]
+domain_info = batch_lookup_proteins(proteins)
+for uid, entries in domain_info.items():
+    print(f"\n{uid}:")
+    for e in entries[:3]:
+        print(f"  - {e['accession']} ({e['type']}): {e['name']}")
+```
+
+### 8. Search by Text or Taxonomy
+
+```python
+def search_entries(query, entry_type=None, taxonomy_id=None):
+    """Search InterPro entries by text."""
+    params = {"search": query, "page_size": 20}
+    if entry_type:
+        params["type"] = entry_type  # family, domain, homologous_superfamily, etc.
+
+    endpoint = "entry/InterPro/"
+    if taxonomy_id:
+        endpoint = f"entry/InterPro/taxonomy/UniProt/{taxonomy_id}/"
+
+    return interpro_get(endpoint, params)
+
+# Search for kinase-related entries
+kinase_entries = search_entries("kinase", entry_type="domain")
+```
+
+## Query Workflows
+
+### Workflow 1: Characterize an Unknown Protein
+
+1. **Run InterProScan** locally or via the web (https://www.ebi.ac.uk/interpro/search/sequence/) to scan a protein sequence
+2. **Parse results** to identify domain architecture
+3. **Look up each InterPro entry** for biological context
+4. **Get GO terms** from associated InterPro entries for functional inference
+
+```python
+# After running InterProScan and getting a UniProt ID:
+def characterize_protein(uniprot_id):
+    """Complete characterization workflow."""
+
+    # 1. Get all annotations
+    entries = get_protein_entries(uniprot_id)
+
+    # 2. Group by type
+    by_type = {}
+    for e in entries.get("results", []):
+        t = e["metadata"]["type"]
+        by_type.setdefault(t, []).append({
+            "accession": e["metadata"]["accession"],
+            "name": e["metadata"]["name"]
+        })
+
+    # 3. Get GO terms
+    go_terms = get_go_terms_for_protein(uniprot_id)
+
+    return {
+        "families": by_type.get("family", []),
+        "domains": by_type.get("domain", []),
+        "superfamilies": by_type.get("homologous_superfamily", []),
+        "go_terms": go_terms
+    }
+```
+
+### Workflow 2: Find All Members of a Protein Family
+
+1. Identify the InterPro family entry ID (e.g., IPR000719 for protein kinases)
+2. Query all UniProt proteins annotated with that entry
+3. Filter by organism/taxonomy if needed
+4. Download FASTA sequences for phylogenetic analysis
+
+### Workflow 3: Comparative Domain Analysis
+
+1. Collect proteins of interest (e.g., all paralogs)
+2. Get domain architecture for each protein
+3. Compare domain compositions and orders
+4. Identify domain gain/loss events
+
+## API Endpoint Summary
+
+| Endpoint | Description |
+|----------|-------------|
+| `/protein/UniProt/{id}/` | Full annotation for a protein |
+| `/protein/UniProt/{id}/entry/InterPro/` | InterPro entries for a protein |
+| `/entry/InterPro/{id}/` | Details of an InterPro entry |
+| `/entry/Pfam/{id}/` | Pfam entry details |
+| `/entry/InterPro/{id}/protein/UniProt/` | Proteins with an entry |
+| `/entry/InterPro/` | Search/list InterPro entries |
+| `/taxonomy/UniProt/{tax_id}/` | Proteins from a taxon |
+| `/structure/PDB/{pdb_id}/` | Structures mapped to InterPro |
+
+## Member Databases
+
+| Database | Focus |
+|----------|-------|
+| Pfam | Protein domains (HMM profiles) |
+| PANTHER | Protein families and subfamilies |
+| PRINTS | Protein fingerprints |
+| ProSitePatterns | Amino acid patterns |
+| ProSiteProfiles | Protein profile patterns |
+| SMART | Protein domain analysis |
+| TIGRFAM | JCVI curated protein families |
+| SUPERFAMILY | Structural classification |
+| CDD | Conserved Domain Database (NCBI) |
+| HAMAP | Microbial protein families |
+| NCBIfam | NCBI curated TIGRFAMs |
+| Gene3D | CATH structural classification |
+| PIRSR | PIR site rules |
+
+## Best Practices
+
+- **Use UniProt accession numbers** (not gene names) for the most reliable lookups
+- **Distinguish types**: `family` gives broad classification; `domain` gives specific structural/functional units
+- **InterProScan is faster for novel sequences**: For sequences not in UniProt, submit to the web service
+- **Handle pagination**: Large result sets require iterating through pages
+- **Combine with UniProt data**: InterPro entries often include links to UniProt, PDB, and GO
+
+## Additional Resources
+
+- **InterPro website**: https://www.ebi.ac.uk/interpro/
+- **InterProScan** (run locally): https://github.com/ebi-pf-team/interproscan
+- **API documentation**: https://github.com/ProteinsWebTeam/interpro7-api/blob/master/docs/
+- **Pfam**: https://www.ebi.ac.uk/interpro/entry/pfam/
+- **Citation**: Paysan-Lafosse T et al. (2023) Nucleic Acids Research. PMID: 36350672