opencode-knowledge-work 0.0.0-stage → 0.1.0

package/content/bio-research/skills/bio-research-instrument-data-to-allotrope/references/flattening_guide.md

@@ -0,0 +1,254 @@
+# Flattening ASM to 2D CSV
+
+Converting hierarchical ASM JSON to flat 2D tables for LIMS import, spreadsheet analysis, or data engineering pipelines.
+
+## Why Flatten?
+
+ASM is semantically rich but hierarchical. Many systems need flat tables:
+- LIMS import (Benchling, STARLIMS, LabWare)
+- Excel/CSV analysis
+- Database loading
+- Quick visual inspection
+
+## Flattening Strategy
+
+### Core Principle
+Each **measurement** becomes one **row**. Metadata is repeated per row.
+
+### What's Excluded
+The flattening intentionally **omits top-level ASM metadata** such as:
+- `$asm.manifest` (model version, schema URIs)
+- Root-level fields outside the technique aggregate document
+
+This keeps the output focused on experimental data. If you need schema version tracking for compliance or audit purposes, consider storing the original ASM JSON alongside the flattened CSV, or modify the flattening script to include these fields.
+
+### Hierarchy to Columns
+```
+ASM Hierarchy                    → Flat Column
+─────────────────────────────────────────────────
+device-system-document.
+  device-identifier              → instrument_serial_number
+  model-number                   → instrument_model
+  
+measurement-aggregate-document.
+  analyst                        → analyst
+  measurement-time               → measurement_datetime
+  
+measurement-document[].
+  sample-identifier              → sample_id
+  viable-cell-density.value      → viable_cell_density
+  viable-cell-density.unit       → viable_cell_density_unit
+  viability.value                → viability_percent
+```
+
+## Column Naming Convention
+
+Use snake_case with descriptive suffixes:
+
+| ASM Field | Flat Column |
+|-----------|-------------|
+| `viable-cell-density` | `viable_cell_density` |
+| `.value` | `_value` (or omit if obvious) |
+| `.unit` | `_unit` |
+| `measurement-time` | `measurement_datetime` |
+
+## Example: Cell Counting
+
+### ASM Input (simplified)
+```json
+{
+  "cell-counting-aggregate-document": {
+    "device-system-document": {
+      "device-identifier": "VCB001",
+      "model-number": "Vi-CELL BLU"
+    },
+    "cell-counting-document": [{
+      "measurement-aggregate-document": {
+        "analyst": "jsmith",
+        "measurement-time": "2024-01-15T10:30:00Z",
+        "measurement-document": [
+          {
+            "sample-identifier": "Sample_A",
+            "viable-cell-density": {"value": 2500000, "unit": "(cell/mL)"},
+            "viability": {"value": 95.2, "unit": "%"}
+          },
+          {
+            "sample-identifier": "Sample_B",
+            "viable-cell-density": {"value": 1800000, "unit": "(cell/mL)"},
+            "viability": {"value": 88.7, "unit": "%"}
+          }
+        ]
+      }
+    }]
+  }
+}
+```
+
+### Flattened Output
+```csv
+sample_id,viable_cell_density,viable_cell_density_unit,viability_percent,analyst,measurement_datetime,instrument_serial_number,instrument_model
+Sample_A,2500000,(cell/mL),95.2,jsmith,2024-01-15T10:30:00Z,VCB001,Vi-CELL BLU
+Sample_B,1800000,(cell/mL),88.7,jsmith,2024-01-15T10:30:00Z,VCB001,Vi-CELL BLU
+```
+
+## Example: Plate Reader
+
+### ASM Input (simplified)
+```json
+{
+  "plate-reader-aggregate-document": {
+    "plate-reader-document": [{
+      "measurement-aggregate-document": {
+        "plate-identifier": "ELISA_001",
+        "measurement-document": [
+          {"well-location": "A1", "absorbance": {"value": 0.125, "unit": "mAU"}},
+          {"well-location": "A2", "absorbance": {"value": 0.892, "unit": "mAU"}},
+          {"well-location": "A3", "absorbance": {"value": 1.456, "unit": "mAU"}}
+        ]
+      }
+    }]
+  }
+}
+```
+
+### Flattened Output
+```csv
+plate_id,well_position,absorbance,absorbance_unit
+ELISA_001,A1,0.125,mAU
+ELISA_001,A2,0.892,mAU
+ELISA_001,A3,1.456,mAU
+```
+
+## Handling Data Cubes
+
+Data cubes (time series, spectra) need special handling:
+
+### Option 1: Expand to rows
+Each point becomes a row:
+```csv
+sample_id,time_seconds,absorbance
+Sample_A,0,0.100
+Sample_A,60,0.125
+Sample_A,120,0.150
+```
+
+### Option 2: Wide format
+Measurements as columns:
+```csv
+sample_id,abs_0s,abs_60s,abs_120s
+Sample_A,0.100,0.125,0.150
+```
+
+### Option 3: JSON array in cell
+Keep as array (some systems support this):
+```csv
+sample_id,absorbance_timeseries
+Sample_A,"[0.100,0.125,0.150]"
+```
+
+## Standard Column Sets by Technique
+
+### Cell Counting
+```
+sample_id, viable_cell_density, viable_cell_density_unit, total_cell_count,
+viability_percent, average_cell_diameter, average_cell_diameter_unit,
+analyst, measurement_datetime, instrument_serial_number
+```
+
+### Spectrophotometry
+```
+sample_id, wavelength_nm, absorbance, pathlength_cm, concentration,
+concentration_unit, a260_a280_ratio, a260_a230_ratio,
+analyst, measurement_datetime, instrument_serial_number
+```
+
+### Plate Reader / ELISA
+```
+plate_id, well_position, sample_type, sample_id, absorbance, absorbance_unit,
+concentration, concentration_unit, dilution_factor, cv_percent,
+analyst, measurement_datetime, instrument_serial_number
+```
+
+### qPCR
+```
+sample_id, target_name, well_position, ct_value, ct_mean, ct_sd,
+quantity, quantity_unit, amplification_efficiency,
+analyst, measurement_datetime, instrument_serial_number
+```
+
+## Python Implementation
+
+```python
+import json
+import pandas as pd
+
+def flatten_asm(asm_dict, technique="cell-counting"):
+    """
+    Flatten ASM JSON to pandas DataFrame.
+    
+    Args:
+        asm_dict: Parsed ASM JSON
+        technique: ASM technique type
+        
+    Returns:
+        pandas DataFrame with one row per measurement
+    """
+    rows = []
+    
+    # Get aggregate document
+    agg_key = f"{technique}-aggregate-document"
+    agg_doc = asm_dict.get(agg_key, {})
+    
+    # Extract device info
+    device = agg_doc.get("device-system-document", {})
+    device_info = {
+        "instrument_serial_number": device.get("device-identifier"),
+        "instrument_model": device.get("model-number")
+    }
+    
+    # Get technique documents
+    doc_key = f"{technique}-document"
+    for doc in agg_doc.get(doc_key, []):
+        meas_agg = doc.get("measurement-aggregate-document", {})
+        
+        # Extract common metadata
+        common = {
+            "analyst": meas_agg.get("analyst"),
+            "measurement_datetime": meas_agg.get("measurement-time"),
+            **device_info
+        }
+        
+        # Extract each measurement
+        for meas in meas_agg.get("measurement-document", []):
+            row = {**common}
+            
+            # Flatten measurement fields
+            for key, value in meas.items():
+                if isinstance(value, dict) and "value" in value:
+                    # Value datum pattern
+                    col = key.replace("-", "_")
+                    row[col] = value["value"]
+                    if "unit" in value:
+                        row[f"{col}_unit"] = value["unit"]
+                else:
+                    row[key.replace("-", "_")] = value
+            
+            rows.append(row)
+    
+    return pd.DataFrame(rows)
+
+# Usage
+with open("asm_output.json") as f:
+    asm = json.load(f)
+
+df = flatten_asm(asm, "cell-counting")
+df.to_csv("flattened_output.csv", index=False)
+```
+
+## LIMS Import Considerations
+
+When importing flattened data into a LIMS:
+- Match column names to your LIMS schema field names
+- Use ISO 8601 date format for timestamps
+- Ensure sample IDs match existing LIMS sample identifiers
+- Check if your LIMS expects units in separate columns or embedded in values

package/content/bio-research/skills/bio-research-instrument-data-to-allotrope/references/supported_instruments.md

@@ -0,0 +1,151 @@
+# Supported Instruments
+
+## What Can This Skill Convert?
+
+**Any instrument data that maps to an Allotrope schema can be converted.** The skill uses a tiered parsing approach:
+
+1. **Native allotropy parsers** (listed below) - Highest fidelity, validated against vendor-specific formats
+2. **Flexible fallback parser** - Handles any tabular data (CSV, Excel, TXT) by mapping columns to ASM fields
+3. **PDF extraction** - Extracts tables from PDFs, then applies flexible parsing
+
+If your instrument isn't listed below, the skill can still convert it as long as your data contains recognizable measurement fields (sample IDs, values, units, timestamps, etc.) that map to an ASM technique schema.
+
+---
+
+## Instruments with Native Allotropy Parsers
+
+The following instruments have optimized parsers in the allotropy library with their Vendor enum values.
+
+## Cell Counting
+
+| Instrument | Vendor Enum | File Types |
+|------------|-------------|------------|
+| Beckman Coulter Vi-CELL BLU | `BECKMAN_VI_CELL_BLU` | .csv |
+| Beckman Coulter Vi-CELL XR | `BECKMAN_VI_CELL_XR` | .txt, .xls, .xlsx |
+| ChemoMetec NucleoView NC-200 | `CHEMOMETEC_NUCLEOVIEW` | .xlsx |
+| ChemoMetec NC-View | `CHEMOMETEC_NC_VIEW` | .xlsx |
+| Revvity Matrix | `REVVITY_MATRIX` | .csv |
+
+## Spectrophotometry (UV-Vis)
+
+| Instrument | Vendor Enum | File Types |
+|------------|-------------|------------|
+| Thermo Fisher NanoDrop One | `THERMO_FISHER_NANODROP_ONE` | .csv, .xlsx |
+| Thermo Fisher NanoDrop Eight | `THERMO_FISHER_NANODROP_EIGHT` | .tsv, .txt |
+| Thermo Fisher NanoDrop 8000 | `THERMO_FISHER_NANODROP_8000` | .csv |
+| Unchained Labs Lunatic | `UNCHAINED_LABS_LUNATIC` | .csv, .xlsx |
+| Thermo Fisher Genesys 30 | `THERMO_FISHER_GENESYS30` | .csv |
+
+## Plate Readers (Multi-mode, Absorbance, Fluorescence)
+
+| Instrument | Vendor Enum | File Types |
+|------------|-------------|------------|
+| Molecular Devices SoftMax Pro | `MOLDEV_SOFTMAX_PRO` | .txt |
+| PerkinElmer EnVision | `PERKIN_ELMER_ENVISION` | .csv |
+| Agilent Gen5 (BioTek) | `AGILENT_GEN5` | .xlsx |
+| Agilent Gen5 Image | `AGILENT_GEN5_IMAGE` | .xlsx |
+| BMG MARS (CLARIOstar) | `BMG_MARS` | .csv, .txt |
+| BMG LabTech Smart Control | `BMG_LABTECH_SMART_CONTROL` | .csv |
+| Thermo SkanIt | `THERMO_SKANIT` | .xlsx |
+| Revvity Kaleido | `REVVITY_KALEIDO` | .csv |
+| Tecan Magellan | `TECAN_MAGELLAN` | .xlsx |
+
+## ELISA / Immunoassay
+
+| Instrument | Vendor Enum | File Types |
+|------------|-------------|------------|
+| Molecular Devices SoftMax Pro | `MOLDEV_SOFTMAX_PRO` | .txt |
+| MSD Discovery Workbench | `MSD_WORKBENCH` | .txt |
+| MSD Methodical Mind | `METHODICAL_MIND` | .xlsx |
+| BMG MARS | `BMG_MARS` | .csv, .txt |
+
+## qPCR / PCR
+
+| Instrument | Vendor Enum | File Types |
+|------------|-------------|------------|
+| Applied Biosystems QuantStudio | `APPBIO_QUANTSTUDIO` | .xlsx |
+| Applied Biosystems QuantStudio Design & Analysis | `APPBIO_QUANTSTUDIO_DESIGNANALYSIS` | .xlsx, .csv |
+| Bio-Rad CFX Maestro | `BIORAD_CFX_MAESTRO` | .csv, .xlsx |
+| Roche LightCycler | `ROCHE_LIGHTCYCLER` | .txt |
+
+## Chromatography (HPLC, LC)
+
+| Instrument | Vendor Enum | File Types |
+|------------|-------------|------------|
+| Waters Empower | `WATERS_EMPOWER` | .xml |
+| Thermo Fisher Chromeleon | `THERMO_FISHER_CHROMELEON` | .xml |
+| Agilent ChemStation | `AGILENT_CHEMSTATION` | .csv |
+
+## Electrophoresis
+
+| Instrument | Vendor Enum | File Types |
+|------------|-------------|------------|
+| Agilent TapeStation | `AGILENT_TAPESTATION` | .csv |
+| PerkinElmer LabChip | `PERKIN_ELMER_LABCHIP` | .csv |
+
+## Flow Cytometry
+
+| Instrument | Vendor Enum | File Types |
+|------------|-------------|------------|
+| BD Biosciences FACSDiva | `BD_BIOSCIENCES_FACSDIVA` | .xml |
+| FlowJo | `FLOWJO` | .wsp |
+
+## Solution Analysis
+
+| Instrument | Vendor Enum | File Types |
+|------------|-------------|------------|
+| Roche Cedex BioHT | `ROCHE_CEDEX_BIOHT` | .xlsx |
+| Beckman Coulter Biomek | `BECKMAN_COULTER_BIOMEK` | .csv |
+
+## Auto-Detection Patterns
+
+The skill attempts to identify instrument type from file contents using these patterns:
+
+### Vi-CELL BLU
+- Column headers: "Sample ID", "Viable cells (x10^6 cells/mL)", "Viability (%)"
+- File structure: CSV with specific column order
+
+### Vi-CELL XR
+- Column headers: "Sample", "Total cells/ml", "Viable cells/ml"
+- Multiple export formats supported
+
+### NanoDrop
+- Column headers: "Sample Name", "Nucleic Acid Conc.", "A260", "A280"
+- 260/280 and 260/230 ratio columns
+
+### Plate Readers (General)
+- Well identifiers (A1-H12 pattern)
+- "Plate", "Well", "Sample" columns
+- Block-based structure with metadata headers
+
+### ELISA
+- Standard curve data with concentrations
+- OD/absorbance readings
+- Sample/blank/standard classification
+
+## Using Vendor Enums
+
+```python
+from allotropy.parser_factory import Vendor
+from allotropy.to_allotrope import allotrope_from_file
+
+# List all supported vendors
+for v in Vendor:
+    print(f"{v.name}: {v.value}")
+
+# Convert file
+asm = allotrope_from_file("data.csv", Vendor.BECKMAN_VI_CELL_BLU)
+```
+
+## Checking Supported Status
+
+```python
+from allotropy.parser_factory import get_parser
+
+# Check if a vendor/file combo is supported
+try:
+    parser = get_parser(Vendor.BECKMAN_VI_CELL_BLU)
+    print("Supported!")
+except Exception as e:
+    print(f"Not supported: {e}")
+```

package/content/bio-research/skills/bio-research-instrument-data-to-allotrope/requirements.txt

@@ -0,0 +1,26 @@
+# Instrument Data to Allotrope Skill - Pinned Dependencies
+#
+# These versions are pinned for reproducibility and determinism.
+# All scientists using this skill should install these exact versions
+# to ensure identical ASM output from the same input files.
+#
+# Installation:
+#   pip install -r requirements.txt --break-system-packages
+#
+# Note: Versions pinned as of 2025-01-05
+
+# Core parsing library - provides native instrument parsers
+allotropy==0.1.55
+
+# Data manipulation and file reading
+pandas==2.0.3
+
+# Excel file support (required by pandas for .xlsx files)
+openpyxl==3.1.2
+
+# PDF parsing support (for instruments that export PDFs)
+pdfplumber==0.9.0
+
+# Scientific computing (optional, but recommended for advanced analysis)
+# numpy==1.24.3  # Uncomment if needed
+# scipy==1.11.1  # Uncomment if needed