@synsci/cli-darwin-x64-baseline 1.1.77 → 1.1.78

package/bin/skills/matchms/references/filtering.md

@@ -0,0 +1,288 @@
+# Matchms Filtering Functions Reference
+
+This document provides a comprehensive reference of all filtering functions available in matchms for processing mass spectrometry data.
+
+## Metadata Processing Filters
+
+### Compound & Chemical Information
+
+**add_compound_name(spectrum)**
+- Adds compound name to the correct metadata field
+- Standardizes compound name storage location
+
+**clean_compound_name(spectrum)**
+- Removes frequently seen unwanted additions from compound names
+- Cleans up formatting inconsistencies
+
+**derive_adduct_from_name(spectrum)**
+- Extracts adduct information from compound names
+- Moves adduct notation to proper metadata field
+
+**derive_formula_from_name(spectrum)**
+- Detects chemical formulas in compound names
+- Relocates formulas to appropriate metadata field
+
+**derive_annotation_from_compound_name(spectrum)**
+- Retrieves SMILES/InChI from PubChem using compound name
+- Automatically annotates chemical structures
+
+### Chemical Structure Conversions
+
+**derive_inchi_from_smiles(spectrum)**
+- Generates InChI from SMILES strings
+- Requires rdkit library
+
+**derive_inchikey_from_inchi(spectrum)**
+- Computes InChIKey from InChI
+- 27-character hashed identifier
+
+**derive_smiles_from_inchi(spectrum)**
+- Creates SMILES from InChI representation
+- Requires rdkit library
+
+**repair_inchi_inchikey_smiles(spectrum)**
+- Corrects misplaced chemical identifiers
+- Fixes metadata field confusion
+
+**repair_not_matching_annotation(spectrum)**
+- Ensures consistency between SMILES, InChI, and InChIKey
+- Validates chemical structure annotations match
+
+**add_fingerprint(spectrum, fingerprint_type="daylight", nbits=2048, radius=2)**
+- Generates molecular fingerprints for similarity calculations
+- Fingerprint types: "daylight", "morgan1", "morgan2", "morgan3"
+- Used with FingerprintSimilarity scoring
+
+### Mass & Charge Information
+
+**add_precursor_mz(spectrum)**
+- Normalizes precursor m/z values
+- Standardizes precursor mass metadata
+
+**add_parent_mass(spectrum, estimate_from_adduct=True)**
+- Calculates neutral parent mass from precursor m/z and adduct
+- Can estimate from adduct if not directly available
+
+**correct_charge(spectrum)**
+- Aligns charge values with ionmode
+- Ensures charge sign matches ionization mode
+
+**make_charge_int(spectrum)**
+- Converts charge to integer format
+- Standardizes charge representation
+
+**clean_adduct(spectrum)**
+- Standardizes adduct notation
+- Corrects common adduct formatting issues
+
+**interpret_pepmass(spectrum)**
+- Parses pepmass field into component values
+- Extracts precursor m/z and intensity from combined field
+
+### Ion Mode & Validation
+
+**derive_ionmode(spectrum)**
+- Determines ionmode from adduct information
+- Infers positive/negative mode from adduct type
+
+**require_correct_ionmode(spectrum, ion_mode)**
+- Filters spectra by specified ionmode
+- Returns None if ionmode doesn't match
+- Use: `spectrum = require_correct_ionmode(spectrum, "positive")`
+
+**require_precursor_mz(spectrum, minimum_accepted_mz=0.0)**
+- Validates precursor m/z presence and value
+- Returns None if missing or below threshold
+
+**require_precursor_below_mz(spectrum, maximum_accepted_mz=1000.0)**
+- Enforces maximum precursor m/z limit
+- Returns None if precursor exceeds threshold
+
+### Retention Information
+
+**add_retention_time(spectrum)**
+- Harmonizes retention time as float values
+- Standardizes RT metadata field
+
+**add_retention_index(spectrum)**
+- Stores retention index in standardized field
+- Normalizes RI metadata
+
+### Data Harmonization
+
+**harmonize_undefined_inchi(spectrum, undefined="", aliases=None)**
+- Standardizes undefined/empty InChI entries
+- Replaces various "unknown" representations with consistent value
+
+**harmonize_undefined_inchikey(spectrum, undefined="", aliases=None)**
+- Standardizes undefined/empty InChIKey entries
+- Unifies missing data representation
+
+**harmonize_undefined_smiles(spectrum, undefined="", aliases=None)**
+- Standardizes undefined/empty SMILES entries
+- Consistent handling of missing structural data
+
+### Repair & Quality Functions
+
+**repair_adduct_based_on_smiles(spectrum, mass_tolerance=0.1)**
+- Corrects adduct using SMILES and mass matching
+- Validates adduct matches calculated mass
+
+**repair_parent_mass_is_mol_wt(spectrum, mass_tolerance=0.1)**
+- Converts molecular weight to monoisotopic mass
+- Fixes common metadata confusion
+
+**repair_precursor_is_parent_mass(spectrum)**
+- Fixes swapped precursor/parent mass values
+- Corrects field misassignments
+
+**repair_smiles_of_salts(spectrum, mass_tolerance=0.1)**
+- Removes salt components to match parent mass
+- Extracts relevant molecular fragment
+
+**require_parent_mass_match_smiles(spectrum, mass_tolerance=0.1)**
+- Validates parent mass against SMILES-calculated mass
+- Returns None if masses don't match within tolerance
+
+**require_valid_annotation(spectrum)**
+- Ensures complete, consistent chemical annotations
+- Validates SMILES, InChI, and InChIKey presence and consistency
+
+## Peak Processing Filters
+
+### Normalization & Selection
+
+**normalize_intensities(spectrum)**
+- Scales peak intensities to unit height (max = 1.0)
+- Essential preprocessing step for similarity calculations
+
+**select_by_intensity(spectrum, intensity_from=0.0, intensity_to=1.0)**
+- Retains peaks within specified absolute intensity range
+- Filters by raw intensity values
+
+**select_by_relative_intensity(spectrum, intensity_from=0.0, intensity_to=1.0)**
+- Keeps peaks within relative intensity bounds
+- Filters as fraction of maximum intensity
+
+**select_by_mz(spectrum, mz_from=0.0, mz_to=1000.0)**
+- Filters peaks by m/z value range
+- Removes peaks outside specified m/z window
+
+### Peak Reduction & Filtering
+
+**reduce_to_number_of_peaks(spectrum, n_max=None, ratio_desired=None)**
+- Removes lowest-intensity peaks when exceeding maximum
+- Can specify absolute number or ratio
+- Use: `spectrum = reduce_to_number_of_peaks(spectrum, n_max=100)`
+
+**remove_peaks_around_precursor_mz(spectrum, mz_tolerance=17)**
+- Eliminates peaks within tolerance of precursor
+- Removes precursor and isotope peaks
+- Common preprocessing for fragment-based similarity
+
+**remove_peaks_outside_top_k(spectrum, k=10, ratio_desired=None)**
+- Retains only peaks near k highest-intensity peaks
+- Focuses on most informative signals
+
+**require_minimum_number_of_peaks(spectrum, n_required=10)**
+- Discards spectra with insufficient peaks
+- Quality control filter
+- Returns None if peak count below threshold
+
+**require_minimum_number_of_high_peaks(spectrum, n_required=5, intensity_threshold=0.05)**
+- Removes spectra lacking high-intensity peaks
+- Ensures data quality
+- Returns None if insufficient peaks above threshold
+
+### Loss Calculation
+
+**add_losses(spectrum, loss_mz_from=5.0, loss_mz_to=200.0)**
+- Derives neutral losses from precursor mass
+- Calculates loss = precursor_mz - fragment_mz
+- Adds losses to spectrum for NeutralLossesCosine scoring
+
+## Pipeline Functions
+
+**default_filters(spectrum)**
+- Applies nine essential metadata filters sequentially:
+  1. make_charge_int
+  2. add_precursor_mz
+  3. add_retention_time
+  4. add_retention_index
+  5. derive_adduct_from_name
+  6. derive_formula_from_name
+  7. clean_compound_name
+  8. harmonize_undefined_smiles
+  9. harmonize_undefined_inchi
+- Recommended starting point for metadata harmonization
+
+**SpectrumProcessor(filters)**
+- Orchestrates multi-filter pipelines
+- Accepts list of filter functions
+- Example:
+```python
+from matchms import SpectrumProcessor
+processor = SpectrumProcessor([
+    default_filters,
+    normalize_intensities,
+    lambda s: select_by_relative_intensity(s, intensity_from=0.01)
+])
+processed = processor(spectrum)
+```
+
+## Common Filter Combinations
+
+### Standard Preprocessing Pipeline
+```python
+from matchms.filtering import (default_filters, normalize_intensities,
+                               select_by_relative_intensity,
+                               require_minimum_number_of_peaks)
+
+spectrum = default_filters(spectrum)
+spectrum = normalize_intensities(spectrum)
+spectrum = select_by_relative_intensity(spectrum, intensity_from=0.01)
+spectrum = require_minimum_number_of_peaks(spectrum, n_required=5)
+```
+
+### Quality Control Pipeline
+```python
+from matchms.filtering import (require_precursor_mz, require_minimum_number_of_peaks,
+                               require_minimum_number_of_high_peaks)
+
+spectrum = require_precursor_mz(spectrum, minimum_accepted_mz=50.0)
+if spectrum is None:
+    # Spectrum failed quality control
+    pass
+spectrum = require_minimum_number_of_peaks(spectrum, n_required=10)
+spectrum = require_minimum_number_of_high_peaks(spectrum, n_required=5)
+```
+
+### Chemical Annotation Pipeline
+```python
+from matchms.filtering import (derive_inchi_from_smiles, derive_inchikey_from_inchi,
+                               add_fingerprint, require_valid_annotation)
+
+spectrum = derive_inchi_from_smiles(spectrum)
+spectrum = derive_inchikey_from_inchi(spectrum)
+spectrum = add_fingerprint(spectrum, fingerprint_type="morgan2", nbits=2048)
+spectrum = require_valid_annotation(spectrum)
+```
+
+### Peak Cleaning Pipeline
+```python
+from matchms.filtering import (normalize_intensities, remove_peaks_around_precursor_mz,
+                               select_by_relative_intensity, reduce_to_number_of_peaks)
+
+spectrum = normalize_intensities(spectrum)
+spectrum = remove_peaks_around_precursor_mz(spectrum, mz_tolerance=17)
+spectrum = select_by_relative_intensity(spectrum, intensity_from=0.01)
+spectrum = reduce_to_number_of_peaks(spectrum, n_max=200)
+```
+
+## Notes on Filter Usage
+
+1. **Order matters**: Apply filters in logical sequence (e.g., normalize before relative intensity selection)
+2. **Filters return None**: Many filters return None for invalid spectra; check for None before proceeding
+3. **Immutability**: Filters typically return modified copies; reassign results to variables
+4. **Pipeline efficiency**: Use SpectrumProcessor for consistent multi-spectrum processing
+5. **Documentation**: For detailed parameters, see matchms.readthedocs.io/en/latest/api/matchms.filtering.html

package/bin/skills/matchms/references/importing_exporting.md

@@ -0,0 +1,416 @@
+# Matchms Importing and Exporting Reference
+
+This document details all file format support in matchms for loading and saving mass spectrometry data.
+
+## Importing Spectra
+
+Matchms provides dedicated functions for loading spectra from various file formats. All import functions return generators for memory-efficient processing of large files.
+
+### Common Import Pattern
+
+```python
+from matchms.importing import load_from_mgf
+
+# Load spectra (returns generator)
+spectra_generator = load_from_mgf("spectra.mgf")
+
+# Convert to list for processing
+spectra = list(spectra_generator)
+```
+
+## Supported Import Formats
+
+### MGF (Mascot Generic Format)
+
+**Function**: `load_from_mgf(filename, metadata_harmonization=True)`
+
+**Description**: Loads spectra from MGF files, a common format for mass spectrometry data exchange.
+
+**Parameters**:
+- `filename` (str): Path to MGF file
+- `metadata_harmonization` (bool, default=True): Apply automatic metadata key harmonization
+
+**Example**:
+```python
+from matchms.importing import load_from_mgf
+
+# Load with metadata harmonization
+spectra = list(load_from_mgf("data.mgf"))
+
+# Load without harmonization
+spectra = list(load_from_mgf("data.mgf", metadata_harmonization=False))
+```
+
+**MGF Format**: Text-based format with BEGIN IONS/END IONS blocks containing metadata and peak lists.
+
+---
+
+### MSP (NIST Mass Spectral Library Format)
+
+**Function**: `load_from_msp(filename, metadata_harmonization=True)`
+
+**Description**: Loads spectra from MSP files, commonly used for spectral libraries.
+
+**Parameters**:
+- `filename` (str): Path to MSP file
+- `metadata_harmonization` (bool, default=True): Apply automatic metadata harmonization
+
+**Example**:
+```python
+from matchms.importing import load_from_msp
+
+spectra = list(load_from_msp("library.msp"))
+```
+
+**MSP Format**: Text-based format with Name/MW/Comment fields followed by peak lists.
+
+---
+
+### mzML (Mass Spectrometry Markup Language)
+
+**Function**: `load_from_mzml(filename, ms_level=2, metadata_harmonization=True)`
+
+**Description**: Loads spectra from mzML files, the standard XML-based format for raw mass spectrometry data.
+
+**Parameters**:
+- `filename` (str): Path to mzML file
+- `ms_level` (int, default=2): MS level to extract (1 for MS1, 2 for MS2/tandem)
+- `metadata_harmonization` (bool, default=True): Apply automatic metadata harmonization
+
+**Example**:
+```python
+from matchms.importing import load_from_mzml
+
+# Load MS2 spectra (default)
+ms2_spectra = list(load_from_mzml("data.mzML"))
+
+# Load MS1 spectra
+ms1_spectra = list(load_from_mzml("data.mzML", ms_level=1))
+```
+
+**mzML Format**: XML-based standard format containing raw instrument data and rich metadata.
+
+---
+
+### mzXML
+
+**Function**: `load_from_mzxml(filename, ms_level=2, metadata_harmonization=True)`
+
+**Description**: Loads spectra from mzXML files, an earlier XML-based format for mass spectrometry data.
+
+**Parameters**:
+- `filename` (str): Path to mzXML file
+- `ms_level` (int, default=2): MS level to extract
+- `metadata_harmonization` (bool, default=True): Apply automatic metadata harmonization
+
+**Example**:
+```python
+from matchms.importing import load_from_mzxml
+
+spectra = list(load_from_mzxml("data.mzXML"))
+```
+
+**mzXML Format**: XML-based format, predecessor to mzML.
+
+---
+
+### JSON (GNPS Format)
+
+**Function**: `load_from_json(filename, metadata_harmonization=True)`
+
+**Description**: Loads spectra from JSON files, particularly GNPS-compatible JSON format.
+
+**Parameters**:
+- `filename` (str): Path to JSON file
+- `metadata_harmonization` (bool, default=True): Apply automatic metadata harmonization
+
+**Example**:
+```python
+from matchms.importing import load_from_json
+
+spectra = list(load_from_json("spectra.json"))
+```
+
+**JSON Format**: Structured JSON with spectrum metadata and peak arrays.
+
+---
+
+### Pickle (Python Serialization)
+
+**Function**: `load_from_pickle(filename)`
+
+**Description**: Loads previously saved matchms Spectrum objects from pickle files. Fast loading of preprocessed spectra.
+
+**Parameters**:
+- `filename` (str): Path to pickle file
+
+**Example**:
+```python
+from matchms.importing import load_from_pickle
+
+spectra = list(load_from_pickle("processed_spectra.pkl"))
+```
+
+**Use case**: Saving and loading preprocessed spectra for faster subsequent analyses.
+
+---
+
+### USI (Universal Spectrum Identifier)
+
+**Function**: `load_from_usi(usi)`
+
+**Description**: Loads a single spectrum from a metabolomics USI reference.
+
+**Parameters**:
+- `usi` (str): Universal Spectrum Identifier string
+
+**Example**:
+```python
+from matchms.importing import load_from_usi
+
+usi = "mzspec:GNPS:TASK-...:spectrum..."
+spectrum = load_from_usi(usi)
+```
+
+**USI Format**: Standardized identifier for accessing spectra from online repositories.
+
+---
+
+## Exporting Spectra
+
+Matchms provides functions to save processed spectra to various formats for sharing and archival.
+
+### MGF Export
+
+**Function**: `save_as_mgf(spectra, filename, write_mode='w')`
+
+**Description**: Saves spectra to MGF format.
+
+**Parameters**:
+- `spectra` (list): List of Spectrum objects to save
+- `filename` (str): Output file path
+- `write_mode` (str, default='w'): File write mode ('w' for write, 'a' for append)
+
+**Example**:
+```python
+from matchms.exporting import save_as_mgf
+
+save_as_mgf(processed_spectra, "output.mgf")
+```
+
+---
+
+### MSP Export
+
+**Function**: `save_as_msp(spectra, filename, write_mode='w')`
+
+**Description**: Saves spectra to MSP format.
+
+**Parameters**:
+- `spectra` (list): List of Spectrum objects to save
+- `filename` (str): Output file path
+- `write_mode` (str, default='w'): File write mode
+
+**Example**:
+```python
+from matchms.exporting import save_as_msp
+
+save_as_msp(library_spectra, "library.msp")
+```
+
+---
+
+### JSON Export
+
+**Function**: `save_as_json(spectra, filename, write_mode='w')`
+
+**Description**: Saves spectra to JSON format (GNPS-compatible).
+
+**Parameters**:
+- `spectra` (list): List of Spectrum objects to save
+- `filename` (str): Output file path
+- `write_mode` (str, default='w'): File write mode
+
+**Example**:
+```python
+from matchms.exporting import save_as_json
+
+save_as_json(spectra, "spectra.json")
+```
+
+---
+
+### Pickle Export
+
+**Function**: `save_as_pickle(spectra, filename)`
+
+**Description**: Saves spectra as Python pickle file. Preserves all Spectrum attributes and is fastest for loading.
+
+**Parameters**:
+- `spectra` (list): List of Spectrum objects to save
+- `filename` (str): Output file path
+
+**Example**:
+```python
+from matchms.exporting import save_as_pickle
+
+save_as_pickle(processed_spectra, "processed.pkl")
+```
+
+**Advantages**:
+- Fast save and load
+- Preserves exact Spectrum state
+- No format conversion overhead
+
+**Disadvantages**:
+- Not human-readable
+- Python-specific (not portable to other languages)
+- Pickle format may not be compatible across Python versions
+
+---
+
+## Complete Import/Export Workflow
+
+### Preprocessing and Saving Pipeline
+
+```python
+from matchms.importing import load_from_mgf
+from matchms.exporting import save_as_mgf, save_as_pickle
+from matchms.filtering import default_filters, normalize_intensities
+from matchms.filtering import select_by_relative_intensity
+
+# Load raw spectra
+spectra = list(load_from_mgf("raw_data.mgf"))
+
+# Process spectra
+processed = []
+for spectrum in spectra:
+    spectrum = default_filters(spectrum)
+    spectrum = normalize_intensities(spectrum)
+    spectrum = select_by_relative_intensity(spectrum, intensity_from=0.01)
+    if spectrum is not None:
+        processed.append(spectrum)
+
+# Save processed spectra (MGF for sharing)
+save_as_mgf(processed, "processed_data.mgf")
+
+# Save as pickle for fast reloading
+save_as_pickle(processed, "processed_data.pkl")
+```
+
+### Format Conversion
+
+```python
+from matchms.importing import load_from_mzml
+from matchms.exporting import save_as_mgf, save_as_msp
+
+# Convert mzML to MGF
+spectra = list(load_from_mzml("data.mzML", ms_level=2))
+save_as_mgf(spectra, "data.mgf")
+
+# Convert to MSP library format
+save_as_msp(spectra, "data.msp")
+```
+
+### Loading from Multiple Files
+
+```python
+from matchms.importing import load_from_mgf
+import glob
+
+# Load all MGF files in directory
+all_spectra = []
+for mgf_file in glob.glob("data/*.mgf"):
+    spectra = list(load_from_mgf(mgf_file))
+    all_spectra.extend(spectra)
+
+print(f"Loaded {len(all_spectra)} spectra from multiple files")
+```
+
+### Memory-Efficient Processing
+
+```python
+from matchms.importing import load_from_mgf
+from matchms.exporting import save_as_mgf
+from matchms.filtering import default_filters, normalize_intensities
+
+# Process large file without loading all into memory
+def process_spectrum(spectrum):
+    spectrum = default_filters(spectrum)
+    spectrum = normalize_intensities(spectrum)
+    return spectrum
+
+# Stream processing
+with open("output.mgf", 'w') as outfile:
+    for spectrum in load_from_mgf("large_file.mgf"):
+        processed = process_spectrum(spectrum)
+        if processed is not None:
+            # Write immediately without storing in memory
+            save_as_mgf([processed], outfile, write_mode='a')
+```
+
+## Format Selection Guidelines
+
+**MGF**:
+- ✓ Widely supported
+- ✓ Human-readable
+- ✓ Good for data sharing
+- ✓ Moderate file size
+- Best for: Data exchange, GNPS uploads, publication data
+
+**MSP**:
+- ✓ Spectral library standard
+- ✓ Human-readable
+- ✓ Good metadata support
+- Best for: Reference libraries, NIST format compatibility
+
+**JSON**:
+- ✓ Structured format
+- ✓ GNPS compatible
+- ✓ Easy to parse programmatically
+- Best for: Web applications, GNPS integration, structured data
+
+**Pickle**:
+- ✓ Fastest save/load
+- ✓ Preserves exact state
+- ✗ Not portable to other languages
+- ✗ Not human-readable
+- Best for: Intermediate processing, Python-only workflows
+
+**mzML/mzXML**:
+- ✓ Raw instrument data
+- ✓ Rich metadata
+- ✓ Industry standard
+- ✗ Large file size
+- ✗ Slower to parse
+- Best for: Raw data archival, multi-level MS data
+
+## Metadata Harmonization
+
+The `metadata_harmonization` parameter (available in most import functions) automatically standardizes metadata keys:
+
+```python
+# Without harmonization
+spectrum = load_from_mgf("data.mgf", metadata_harmonization=False)
+# May have: "PRECURSOR_MZ", "Precursor_mz", "precursormz"
+
+# With harmonization (default)
+spectrum = load_from_mgf("data.mgf", metadata_harmonization=True)
+# Standardized to: "precursor_mz"
+```
+
+**Recommended**: Keep harmonization enabled (default) for consistent metadata access across different data sources.
+
+## File Format Specifications
+
+For detailed format specifications:
+- **MGF**: http://www.matrixscience.com/help/data_file_help.html
+- **MSP**: https://chemdata.nist.gov/mass-spc/ms-search/
+- **mzML**: http://www.psidev.info/mzML
+- **GNPS JSON**: https://gnps.ucsd.edu/
+
+## Further Reading
+
+For complete API documentation:
+https://matchms.readthedocs.io/en/latest/api/matchms.importing.html
+https://matchms.readthedocs.io/en/latest/api/matchms.exporting.html