@synsci/cli-darwin-x64 1.1.97 → 1.1.99

package/bin/skills/exploratory-data-analysis/references/proteomics_metabolomics_formats.md

@@ -1,517 +0,0 @@
-# Proteomics and Metabolomics File Formats Reference
-
-This reference covers file formats specific to proteomics, metabolomics, lipidomics, and related omics workflows.
-
-## Mass Spectrometry-Based Proteomics
-
-### .mzML - Mass Spectrometry Markup Language
-**Description:** Standard XML format for MS data
-**Typical Data:** MS1 and MS2 spectra, retention times, intensities
-**Use Cases:** Proteomics, metabolomics pipelines
-**Python Libraries:**
-- `pymzml`: `pymzml.run.Reader('file.mzML')`
-- `pyteomics.mzml`: `pyteomics.mzml.read('file.mzML')`
-- `pyopenms`: OpenMS Python bindings
-**EDA Approach:**
-- Scan count and MS level distribution
-- Total ion chromatogram (TIC) analysis
-- Base peak chromatogram (BPC)
-- m/z coverage and resolution
-- Retention time range
-- Precursor selection patterns
-- Data completeness
-- Quality control metrics (lock mass, standards)
-
-### .mzXML - Legacy MS XML Format
-**Description:** Older XML-based MS format
-**Typical Data:** Mass spectra with metadata
-**Use Cases:** Legacy proteomics data
-**Python Libraries:**
-- `pyteomics.mzxml`
-- `pymzml`: Can read mzXML
-**EDA Approach:**
-- Similar to mzML
-- Format version compatibility
-- Conversion quality validation
-- Metadata preservation check
-
-### .mzIdentML - Peptide Identification Format
-**Description:** PSI standard for peptide identifications
-**Typical Data:** Peptide-spectrum matches, proteins, scores
-**Use Cases:** Search engine results, proteomics workflows
-**Python Libraries:**
-- `pyteomics.mzid`
-- `pyopenms`: MzIdentML support
-**EDA Approach:**
-- PSM count and score distribution
-- FDR calculation and filtering
-- Modification analysis
-- Missed cleavage statistics
-- Protein inference results
-- Search parameters validation
-- Decoy hit analysis
-- Rank-1 vs lower ranks
-
-### .pepXML - Trans-Proteomic Pipeline Peptide XML
-**Description:** TPP format for peptide identifications
-**Typical Data:** Search results with statistical validation
-**Use Cases:** Proteomics database search output
-**Python Libraries:**
-- `pyteomics.pepxml`
-**EDA Approach:**
-- Search engine comparison
-- Score distributions (XCorr, expect value, etc.)
-- Charge state analysis
-- Modification frequencies
-- PeptideProphet probabilities
-- Protein coverage
-- Spectral counting
-
-### .protXML - Protein Inference Results
-**Description:** TPP protein-level identifications
-**Typical Data:** Protein groups, probabilities, peptides
-**Use Cases:** Protein-level analysis
-**Python Libraries:**
-- `pyteomics.protxml`
-**EDA Approach:**
-- Protein group statistics
-- Parsimonious protein sets
-- ProteinProphet probabilities
-- Coverage and peptide count per protein
-- Unique vs shared peptides
-- Protein molecular weight distribution
-- GO term enrichment preparation
-
-### .pride.xml - PRIDE XML Format
-**Description:** Proteomics Identifications Database format
-**Typical Data:** Complete proteomics experiment data
-**Use Cases:** Public data deposition (legacy)
-**Python Libraries:**
-- `pyteomics.pride`
-- Custom XML parsers
-**EDA Approach:**
-- Experiment metadata extraction
-- Identification completeness
-- Cross-linking to spectra
-- Protocol information
-- Instrument details
-
-### .tsv / .csv (Proteomics)
-**Description:** Tab or comma-separated proteomics results
-**Typical Data:** Peptide or protein quantification tables
-**Use Cases:** MaxQuant, Proteome Discoverer, Skyline output
-**Python Libraries:**
-- `pandas`: `pd.read_csv()` or `pd.read_table()`
-**EDA Approach:**
-- Identification counts
-- Quantitative value distributions
-- Missing value patterns
-- Intensity-based analysis
-- Label-free quantification assessment
-- Isobaric tag ratio analysis
-- Coefficient of variation
-- Batch effects
-
-### .msf - Thermo MSF Database
-**Description:** Proteome Discoverer results database
-**Typical Data:** SQLite database with search results
-**Use Cases:** Thermo Proteome Discoverer workflows
-**Python Libraries:**
-- `sqlite3`: Database access
-- Custom MSF parsers
-**EDA Approach:**
-- Database schema exploration
-- Peptide and protein tables
-- Score thresholds
-- Quantification data
-- Processing node information
-- Confidence levels
-
-### .pdResult - Proteome Discoverer Result
-**Description:** Proteome Discoverer study results
-**Typical Data:** Comprehensive search and quantification
-**Use Cases:** PD study exports
-**Python Libraries:**
-- Vendor tools for conversion
-- Export to TSV for Python analysis
-**EDA Approach:**
-- Study design validation
-- Result filtering criteria
-- Quantitative comparison groups
-- Imputation strategies
-
-### .pep.xml - Peptide Summary
-**Description:** Compact peptide identification format
-**Typical Data:** Peptide sequences, modifications, scores
-**Use Cases:** Downstream analysis input
-**Python Libraries:**
-- `pyteomics`: XML parsing
-**EDA Approach:**
-- Unique peptide counting
-- PTM site localization
-- Retention time predictability
-- Charge state preferences
-
-## Quantitative Proteomics
-
-### .sky - Skyline Document
-**Description:** Skyline targeted proteomics document
-**Typical Data:** Transition lists, chromatograms, results
-**Use Cases:** Targeted proteomics (SRM/MRM/PRM)
-**Python Libraries:**
-- `skyline`: Python API (limited)
-- Export to CSV for analysis
-**EDA Approach:**
-- Transition selection validation
-- Chromatographic peak quality
-- Interference detection
-- Retention time consistency
-- Calibration curve assessment
-- Replicate correlation
-- LOD/LOQ determination
-
-### .sky.zip - Zipped Skyline Document
-**Description:** Skyline document with external files
-**Typical Data:** Complete Skyline analysis
-**Use Cases:** Sharing Skyline projects
-**Python Libraries:**
-- `zipfile`: Extract for processing
-**EDA Approach:**
-- Document structure
-- External file references
-- Result export and analysis
-
-### .wiff - SCIEX WIFF Format
-**Description:** SCIEX instrument data with quantitation
-**Typical Data:** LC-MS/MS with MRM transitions
-**Use Cases:** SCIEX QTRAP, TripleTOF data
-**Python Libraries:**
-- Vendor tools (limited Python access)
-- Conversion to mzML
-**EDA Approach:**
-- MRM transition performance
-- Dwell time optimization
-- Cycle time analysis
-- Peak integration quality
-
-### .raw (Thermo)
-**Description:** Thermo raw instrument file
-**Typical Data:** Full MS data from Orbitrap, Q Exactive
-**Use Cases:** Label-free and TMT quantification
-**Python Libraries:**
-- `pymsfilereader`: Thermo RawFileReader
-- `ThermoRawFileParser`: Cross-platform CLI
-**EDA Approach:**
-- MS1 and MS2 acquisition rates
-- AGC target and fill times
-- Resolution settings
-- Isolation window validation
-- SPS ion selection (TMT)
-- Contamination assessment
-
-### .d (Agilent)
-**Description:** Agilent data directory
-**Typical Data:** LC-MS and GC-MS data
-**Use Cases:** Agilent instrument workflows
-**Python Libraries:**
-- Community parsers
-- Export to mzML
-**EDA Approach:**
-- Method consistency
-- Calibration status
-- Sequence run information
-- Retention time stability
-
-## Metabolomics and Lipidomics
-
-### .mzML (Metabolomics)
-**Description:** Standard MS format for metabolomics
-**Typical Data:** Full scan MS, targeted MS/MS
-**Use Cases:** Untargeted and targeted metabolomics
-**Python Libraries:**
-- Same as proteomics mzML tools
-**EDA Approach:**
-- Feature detection quality
-- Mass accuracy assessment
-- Retention time alignment
-- Blank subtraction
-- QC sample consistency
-- Isotope pattern validation
-- Adduct formation analysis
-- In-source fragmentation check
-
-### .cdf / .netCDF - ANDI-MS
-**Description:** Analytical Data Interchange for MS
-**Typical Data:** GC-MS, LC-MS chromatography data
-**Use Cases:** Metabolomics, GC-MS workflows
-**Python Libraries:**
-- `netCDF4`: Low-level access
-- `pyopenms`: CDF support
-- `xcms` via R integration
-**EDA Approach:**
-- TIC and extracted ion chromatograms
-- Peak detection across samples
-- Retention index calculation
-- Mass spectral matching
-- Library search preparation
-
-### .msp - Mass Spectral Format (NIST)
-**Description:** NIST spectral library format
-**Typical Data:** Reference mass spectra
-**Use Cases:** Metabolite identification, library matching
-**Python Libraries:**
-- `matchms`: Spectral matching
-- Custom MSP parsers
-**EDA Approach:**
-- Library coverage
-- Metadata completeness (InChI, SMILES)
-- Spectral quality metrics
-- Collision energy standardization
-- Precursor type annotation
-
-### .mgf (Metabolomics)
-**Description:** Mascot Generic Format for MS/MS
-**Typical Data:** MS/MS spectra for metabolite ID
-**Use Cases:** Spectral library searching
-**Python Libraries:**
-- `matchms`: Metabolomics spectral analysis
-- `pyteomics.mgf`
-**EDA Approach:**
-- Spectrum quality filtering
-- Precursor isolation purity
-- Fragment m/z accuracy
-- Neutral loss patterns
-- MS/MS completeness
-
-### .nmrML - NMR Markup Language
-**Description:** Standard XML format for NMR metabolomics
-**Typical Data:** 1D/2D NMR spectra with metadata
-**Use Cases:** NMR-based metabolomics
-**Python Libraries:**
-- `nmrml2isa`: Format conversion
-- Custom XML parsers
-**EDA Approach:**
-- Spectral quality metrics
-- Binning consistency
-- Reference compound validation
-- pH and temperature effects
-- Metabolite identification confidence
-
-### .json (Metabolomics)
-**Description:** JSON format for metabolomics results
-**Typical Data:** Feature tables, annotations, metadata
-**Use Cases:** GNPS, MetaboAnalyst, web tools
-**Python Libraries:**
-- `json`: Standard library
-- `pandas`: JSON normalization
-**EDA Approach:**
-- Feature annotation coverage
-- GNPS clustering results
-- Molecular networking statistics
-- Adduct and in-source fragment linkage
-- Putative identification confidence
-
-### .txt (Metabolomics Tables)
-**Description:** Tab-delimited feature tables
-**Typical Data:** m/z, RT, intensities across samples
-**Use Cases:** MZmine, XCMS, MS-DIAL output
-**Python Libraries:**
-- `pandas`: Text file reading
-**EDA Approach:**
-- Feature count and quality
-- Missing value imputation
-- Data normalization assessment
-- Batch correction validation
-- PCA and clustering for QC
-- Fold change calculations
-- Statistical test preparation
-
-### .featureXML - OpenMS Feature Format
-**Description:** OpenMS detected features
-**Typical Data:** LC-MS features with quality scores
-**Use Cases:** OpenMS workflows
-**Python Libraries:**
-- `pyopenms`: FeatureXML support
-**EDA Approach:**
-- Feature detection parameters
-- Quality metrics per feature
-- Isotope pattern fitting
-- Charge state assignment
-- FWHM and asymmetry
-
-### .consensusXML - OpenMS Consensus Features
-**Description:** Linked features across samples
-**Typical Data:** Aligned features with group info
-**Use Cases:** Multi-sample LC-MS analysis
-**Python Libraries:**
-- `pyopenms`: ConsensusXML reading
-**EDA Approach:**
-- Feature correspondence quality
-- Retention time alignment
-- Missing value patterns
-- Intensity normalization needs
-- Batch-wise feature agreement
-
-### .idXML - OpenMS Identification Format
-**Description:** Peptide/metabolite identifications
-**Typical Data:** MS/MS identifications with scores
-**Use Cases:** OpenMS ID workflows
-**Python Libraries:**
-- `pyopenms`: IdXML support
-**EDA Approach:**
-- Identification rate
-- Score distribution
-- Spectral match quality
-- False discovery assessment
-- Annotation transfer validation
-
-## Lipidomics-Specific Formats
-
-### .lcb - LipidCreator Batch
-**Description:** LipidCreator transition list
-**Typical Data:** Lipid transitions for targeted MS
-**Use Cases:** Targeted lipidomics
-**Python Libraries:**
-- Export to CSV for processing
-**EDA Approach:**
-- Transition coverage per lipid class
-- Retention time prediction
-- Collision energy optimization
-- Class-specific fragmentation patterns
-
-### .mzTab - Proteomics/Metabolomics Tabular Format
-**Description:** PSI tabular summary format
-**Typical Data:** Protein/peptide/metabolite quantification
-**Use Cases:** Publication and data sharing
-**Python Libraries:**
-- `pyteomics.mztab`
-- `pandas` for TSV-like structure
-**EDA Approach:**
-- Data completeness
-- Metadata section validation
-- Quantification method
-- Identification confidence
-- Software and parameters
-- Quality metrics summary
-
-### .csv (LipidSearch, LipidMatch)
-**Description:** Lipid identification results
-**Typical Data:** Lipid annotations, grades, intensities
-**Use Cases:** Lipidomics software output
-**Python Libraries:**
-- `pandas`: CSV reading
-**EDA Approach:**
-- Lipid class distribution
-- Identification grade/confidence
-- Fatty acid composition analysis
-- Double bond and chain length patterns
-- Intensity correlations
-- Normalization to internal standards
-
-### .sdf (Metabolomics)
-**Description:** Structure data file for metabolites
-**Typical Data:** Chemical structures with properties
-**Use Cases:** Metabolite database creation
-**Python Libraries:**
-- `RDKit`: `Chem.SDMolSupplier('file.sdf')`
-**EDA Approach:**
-- Structure validation
-- Property calculation (logP, MW, TPSA)
-- Molecular formula consistency
-- Tautomer enumeration
-- Retention time prediction features
-
-### .mol (Metabolomics)
-**Description:** Single molecule structure files
-**Typical Data:** Metabolite chemical structure
-**Use Cases:** Structure-based searches
-**Python Libraries:**
-- `RDKit`: `Chem.MolFromMolFile('file.mol')`
-**EDA Approach:**
-- Structure correctness
-- Stereochemistry validation
-- Charge state
-- Implicit hydrogen handling
-
-## Data Processing and Analysis
-
-### .h5 / .hdf5 (Omics)
-**Description:** HDF5 for large omics datasets
-**Typical Data:** Feature matrices, spectra, metadata
-**Use Cases:** Large-scale studies, cloud computing
-**Python Libraries:**
-- `h5py`: HDF5 access
-- `anndata`: For single-cell proteomics
-**EDA Approach:**
-- Dataset organization
-- Chunking and compression
-- Metadata structure
-- Efficient data access patterns
-- Sample and feature annotations
-
-### .Rdata / .rds - R Objects
-**Description:** Serialized R analysis objects
-**Typical Data:** Processed omics results from R packages
-**Use Cases:** xcms, CAMERA, MSnbase workflows
-**Python Libraries:**
-- `pyreadr`: `pyreadr.read_r('file.Rdata')`
-- `rpy2`: R-Python integration
-**EDA Approach:**
-- Object structure exploration
-- Data extraction
-- Method parameter review
-- Conversion to Python-native formats
-
-### .mzTab-M - Metabolomics mzTab
-**Description:** mzTab specific to metabolomics
-**Typical Data:** Small molecule quantification
-**Use Cases:** Metabolomics data sharing
-**Python Libraries:**
-- `pyteomics.mztab`: Can parse mzTab-M
-**EDA Approach:**
-- Small molecule evidence
-- Feature quantification
-- Database references (HMDB, KEGG, etc.)
-- Adduct and charge annotation
-- MS level information
-
-### .parquet (Omics)
-**Description:** Columnar storage for large tables
-**Typical Data:** Feature matrices, metadata
-**Use Cases:** Efficient big data omics
-**Python Libraries:**
-- `pandas`: `pd.read_parquet()`
-- `pyarrow`: Direct parquet access
-**EDA Approach:**
-- Compression efficiency
-- Column-wise statistics
-- Partition structure
-- Schema validation
-- Fast filtering and aggregation
-
-### .pkl (Omics Models)
-**Description:** Pickled Python objects
-**Typical Data:** ML models, processed data
-**Use Cases:** Workflow intermediate storage
-**Python Libraries:**
-- `pickle`: Standard serialization
-- `joblib`: Enhanced pickling
-**EDA Approach:**
-- Object type and structure
-- Model parameters
-- Feature importance (if ML model)
-- Data shapes and types
-- Deserialization validation
-
-### .zarr (Omics)
-**Description:** Chunked, compressed array storage
-**Typical Data:** Multi-dimensional omics data
-**Use Cases:** Cloud-optimized analysis
-**Python Libraries:**
-- `zarr`: Array storage
-**EDA Approach:**
-- Chunk optimization
-- Compression codecs
-- Multi-scale data
-- Parallel access patterns
-- Metadata annotations