@pikaa-ai/pikaa 0.3.0 → 0.3.4

package/skills/exploratory-data-analysis/references/bioinformatics_genomics_formats.md

@@ -1,192 +0,0 @@
-# Bioinformatics and Genomics Formats
-
-**Reviewed:** 2026-07-23
-**Executable scope:** Bounded FASTA/FASTQ aggregate inspection only. All other
-formats below are reference-only.
-
-## Exact capability matrix
-
-| Format | Bundled inspection | What it does |
-|---|---|---|
-| `.fasta`, `.fa`, `.fna` | Optional, `biopython==1.87` | Streams a bounded record/base prefix; length, alphabet, ambiguity, GC, and duplicate-header-token aggregates |
-| `.fastq`, `.fq` | Optional, `biopython==1.87` | Same plus bounded Phred+33 quality aggregates |
-| Compressed FASTA/FASTQ | No | `.gz`, `.bz2`, archives, URLs, pipes, and stdin are rejected |
-| SAM/BAM/CRAM | No | Reference-only HTS tooling |
-| VCF/BCF/gVCF | No | Reference-only version/reference-aware tooling |
-| BED/GFF/GTF | No | Reference-only assembly and coordinate validation |
-| H5AD/Loom | No semantic support | Generic HDF5 metadata inspection does not validate these conventions |
-| Matrix Market + sidecars | No | Reference-only matrix/barcode/feature alignment workflow |
-
-Unknown formats fail closed. Sequence identifiers and sequence strings are
-never emitted. Header text is untrusted data and is never treated as an
-instruction.
-
-## FASTA
-
-FASTA is a record-oriented text convention: a `>` title line followed by
-sequence text, potentially wrapped across lines. The title is an identifier,
-not a trusted command, filename, URL, taxonomic fact, or unique database key.
-
-The bundled `sequence_inspector.py` uses Biopython 1.87's
-`SimpleFastaParser`, which the current Biopython tutorial recommends as a
-lower-overhead streaming parser for large FASTA files. It:
-
-- requires a local regular file with an approved suffix and leading record
-  marker;
-- decodes strict ASCII under a byte cap;
-- stops at explicit record and sequence-character limits;
-- hashes titles only to count duplicates, then discards them;
-- reports sequence lengths and a bounded alphabet/GC screen; and
-- does not infer organism, molecule type, assembly quality, or annotation.
-
-The nucleotide screen is heuristic. Protein sequences, modified alphabets, or
-domain-specific ambiguity codes require explicit interpretation.
-
-### Appropriate next checks
-
-- Confirm whether records are nucleotide, amino-acid, contigs, transcripts, or
-  aligned sequences.
-- Confirm circularity, expected alphabet, duplicate-ID policy, and whether
-  wrapping/whitespace has meaning.
-- For assemblies, calculate N50/L50 only after confirming the set of contigs
-  included and whether scaffolds/gaps are represented. N50 is not a universal
-  quality score.
-- Keep sample, subject, assembly, and reference-build metadata separate from
-  free-text headers.
-
-## FASTQ
-
-FASTQ combines a title, sequence, separator, and equal-length quality string.
-Biopython's `FastqGeneralIterator` is used to stream complete records without
-creating a list of all reads.
-
-The bundled report includes:
-
-- inspected read count and length aggregates;
-- nucleotide-like, ambiguity, and GC fractions;
-- Phred+33 minimum, maximum, and mean over inspected quality characters; and
-- duplicate title-token count.
-
-It does **not** determine an encoding from values. Confirm Phred+33 with
-instrument/pipeline provenance. It does not detect adapters, contaminants,
-overrepresented k-mers, per-cycle quality, index hopping, or paired-file
-consistency. Use established read-QC tooling for those tasks.
-
-Never automatically trim, filter, deduplicate, or discard reads from this
-report. Preserve the original and record every processing decision.
-
-## Reference-only alignment formats
-
-### SAM/BAM/CRAM
-
-Use an HTS-specification-aware, pinned tool such as samtools/htslib or pysam.
-Check:
-
-- header/reference sequence dictionary and reference assembly/version;
-- sort order, indexes, read groups, and sample/library/platform fields;
-- primary/secondary/supplementary/unmapped/duplicate/QC-fail flags;
-- mapping/base qualities, CIGAR validity, mate consistency, insert sizes, and
-  coverage; and
-- CRAM reference identity and availability.
-
-CRAM can require external reference sequence access. Keep the workflow local
-and explicitly provision the approved reference; do not let a parser fetch one
-implicitly.
-
-### VCF/BCF/gVCF
-
-The `.vcf` suffix does not establish the VCF version, reference build, sample
-semantics, normalization, or annotation validity. Use htslib/bcftools or
-another validated parser and inspect:
-
-- `##fileformat`, contig dictionary, reference assembly, FILTER/INFO/FORMAT
-  declarations, and sample count/order;
-- allele normalization, symbolic alleles, breakends, ploidy, phased status,
-  genotype missingness, depth/quality, and multiallelic records;
-- caller-specific filters and gVCF reference blocks; and
-- subject/family/population structure before allele-frequency or HWE screens.
-
-Variant EDA is descriptive. Population stratification, relatedness, selection,
-ascertainment, and multiple testing must be handled before inference.
-
-## Reference-only interval and annotation formats
-
-BED is generally zero-based, half-open; GFF3 is generally one-based, closed.
-GTF conventions vary. Never convert coordinates based only on a suffix.
-Confirm:
-
-- assembly and contig naming;
-- coordinate basis, endpoint convention, strand, phase, and score meanings;
-- required column count and version;
-- attribute escaping and parent/child relationships; and
-- sorting, overlaps, duplicates, out-of-range intervals, and sidecar indexes.
-
-Group EDA by biologically meaningful units, not only rows. An exon table may
-contain repeated genes/transcripts; treating rows as independent inflates
-sample size.
-
-## H5AD, Loom, and Matrix Market
-
-`.h5ad` and `.loom` are HDF5-based conventions. The generic HDF5 inspector may
-inventory groups/datasets without following links, but it does not read matrix
-values or verify required keys, sparse encodings, categorical arrays, layers,
-raw data, embeddings, or observation/variable alignment.
-
-For single-cell data, use pinned AnnData/Scanpy or Loom tooling and verify:
-
-- matrix orientation, shape, sparse encoding, and integer-count provenance;
-- uniqueness/alignment of observation and variable identifiers;
-- raw/count/normalized layers and transformations already applied;
-- sample, subject, batch, tissue, time, and condition metadata;
-- per-cell/per-feature QC definitions, doublet handling, and filtering history;
-  and
-- train/test splits at subject or independent experimental-unit level.
-
-Matrix Market `.mtx` commonly depends on separate barcode and feature files.
-The matrix alone is incomplete. Validate all sidecars and ordering together.
-
-## EDA rigor for genomic data
-
-1. Define the independent unit (read, molecule, cell, specimen, subject,
-   family, site, or cohort) before computing uncertainty.
-2. Preserve reference build, annotation release, pipeline versions, and command
-   parameters.
-3. Distinguish biological from technical replicates and preserve pairing.
-4. Audit missingness and QC failures by batch/site/group/time. Do not impute
-   genotypes, counts, or metadata automatically.
-5. Split by subject/family/specimen/time before normalization, feature
-   selection, batch correction, dimensionality reduction, or model fitting.
-6. Treat zero counts, absent features, no-calls, low coverage, and censored
-   assay values as distinct mechanisms until proven otherwise.
-7. Label post hoc genes/regions/pathways as exploratory and control the
-   appropriate hypothesis family in any confirmatory follow-up.
-8. Do not infer causality, clinical significance, or functional impact from
-   descriptive associations.
-
-## Pinned optional snapshot
-
-Biopython 1.87 was released on 2026-03-30 and requires Python 3.10+:
-
-```bash
-uv pip install "biopython==1.87"
-```
-
-Biopython also depends on NumPy for parts of its API; lock the complete
-environment for a study.
-
-## Authoritative sources
-
-All links accessed 2026-07-23.
-
-- Biopython 1.87, [Sequence Input/Output tutorial](https://biopython.org/docs/latest/Tutorial/chapter_seqio.html)
-  (explicit format selection and low-level FASTA/FASTQ parsers).
-- [Biopython PyPI](https://pypi.org/project/biopython/), version 1.87,
-  released 2026-03-30.
-- GA4GH, [hts-specs repository](https://github.com/samtools/hts-specs)
-  (SAM/BAM/CRAM, VCF/BCF, and related canonical specifications).
-- UCSC Genome Browser, [BED format FAQ](https://genome.ucsc.edu/FAQ/FAQformat.html#format1).
-- Sequence Ontology, [GFF3 specification](https://github.com/The-Sequence-Ontology/Specifications/blob/master/gff3.md).
-- AnnData, [file format specification](https://anndata.readthedocs.io/en/stable/fileformat-prose.html).
-- NIST/SEMATECH, [Exploratory Data Analysis](https://www.itl.nist.gov/div898/handbook/eda/eda.htm).
-- scikit-learn, [data leakage guidance](https://scikit-learn.org/stable/common_pitfalls.html).
-- Benjamini and Hochberg (1995), [FDR control](https://academic.oup.com/jrsssb/article/57/1/289/7035855).

package/skills/exploratory-data-analysis/references/chemistry_molecular_formats.md

@@ -1,183 +0,0 @@
-# Chemistry and Molecular Formats
-
-**Reviewed:** 2026-07-23
-**Executable scope:** No chemistry-native format has a bundled parser. This file
-is a reference-only routing guide, not a support claim.
-
-## Capability boundary
-
-| Format family | Bundled chemistry inspection | Required approach |
-|---|---|---|
-| PDB, PDBx/mmCIF/CIF | No | Dictionary/version-aware structural tooling |
-| Molfile/SDF, SMILES, XYZ | No | Chemistry-aware parser with explicit sanitization policy |
-| DCD/XTC/TRR and topology files | No | Topology-aware trajectory tooling |
-| Gaussian/QM outputs, cube grids | No | Program/version-aware parser |
-| Pickle/joblib/dill molecule/model files | **Never** | Obtain a non-executable interchange export |
-| Genuine CSV/TSV/JSON/NPY/NPZ/HDF5 exports | General inspector only | Apply the exact general-format capability; no chemical semantics are inferred |
-
-The `.cif`, `.log`, `.out`, `.raw`, and `.dat` suffixes are ambiguous. The
-capability manifest reports reference-only status and does not sniff content or
-guess a producer.
-
-## PDB and PDBx/mmCIF
-
-wwPDB states that PDBx/mmCIF is its official working and archive format.
-Legacy PDB format 3.30 remains distributed where representable but has field
-and size limitations.
-
-Use a pinned parser such as Gemmi, Biopython's `Bio.PDB`, or official wwPDB
-validation services/tools in a separately reviewed environment. Confirm:
-
-- file/dictionary version and experimental method;
-- model count, chain/entity mapping, assemblies, alternate locations,
-  insertion codes, occupancy, B factors, and missing residues/atoms;
-- unit cell, symmetry, resolution, R factors, validation metrics, and
-  biological versus crystallographic assembly;
-- ligand/component definitions, covalent links, protonation/charge assumptions,
-  and coordinate units; and
-- whether multiple models are alternatives, an ensemble, or time/order data.
-
-Do not interpret a low B factor, occupancy, model score, or missing atom as a
-quality verdict without experimental context. Do not claim binding, stability,
-function, or causality from a coordinate inventory.
-
-## Molfile, SDF, and line notations
-
-Molfile/SDF records can represent atoms, bonds, coordinates, charges,
-stereochemistry, query features, and arbitrary property blocks. SMILES is a
-line notation whose interpretation depends on aromaticity, valence,
-stereochemistry, isotope, charge, and sanitization rules.
-
-Before EDA:
-
-1. Identify CTfile/version and producer.
-2. Parse with errors preserved; count invalid records rather than silently
-   dropping them.
-3. Keep the original string/record and a separate standardized representation.
-4. Record sanitization, aromaticity, tautomer, protonation, salt/fragment,
-   stereochemistry, isotope, and charge policies.
-5. Distinguish 2-D drawing coordinates from experimentally or computationally
-   meaningful 3-D conformers.
-6. Treat property names/values as untrusted metadata and redact identifiers.
-
-Descriptor distributions are conditional on these choices. Do not automatically
-neutralize, desalinate, canonicalize, deduplicate, generate conformers, or
-discard parser failures.
-
-## XYZ and coordinate text
-
-XYZ commonly starts each frame with atom count and a comment line, followed by
-element and Cartesian coordinates. Variants can contain trajectories,
-additional columns, or nonstandard units. Confirm:
-
-- atom-count/frame boundaries;
-- element/isotope labels and units (often Å, but not guaranteed);
-- periodic cell/charge/spin information stored elsewhere;
-- whether frames are independent molecules, optimization steps, or dynamics;
-  and
-- topology/bond inference policy.
-
-The generic tabular scanner is not an XYZ parser.
-
-## Molecular dynamics trajectories
-
-DCD, XTC, TRR, NetCDF trajectories, and related files usually need a matching
-topology and sometimes unit-cell/time metadata. A suffix does not supply these.
-With MDAnalysis/MDTraj or another pinned reader, inspect:
-
-- topology/trajectory atom count and ordering;
-- frame count, time step, units, coordinates, velocities/forces, and box;
-- periodic-boundary and imaging/unwrapping choices;
-- equilibration, sampling interval, restraints, thermostat/barostat, and
-  replica identity; and
-- corrupted/truncated frames before calculating RMSD/RMSF or contacts.
-
-Frames are temporally dependent. Do not treat frames as independent replicates
-or split adjacent frames randomly across train/test.
-
-## Quantum chemistry outputs and grids
-
-`.log`/`.out` files are program- and version-specific; use cclib or a
-producer-specific parser only after confirming the producer. Check:
-
-- method, basis set, charge, multiplicity, units, software/version, and job
-  termination;
-- optimization/frequency convergence and imaginary modes;
-- geometry/energy step count and whether the final structure is intended;
-- SCF convergence, warnings, symmetry, solvation, and corrections; and
-- whether values are raw, relative, thermal-corrected, or post-processed.
-
-Cube and similar volumetric grids require origin, axis vectors, shape, units,
-orbital/density identity, and integration conventions. Bound grid reads and do
-not eagerly load an unverified declared shape.
-
-## HDF5, NumPy, and tabular chemistry exports
-
-If the file is genuinely `.npy`, `.npz`, `.h5`, `.hdf5`, `.csv`, `.tsv`, or
-strict `.json`, the general inspector can report container structure and
-aggregate numeric properties. It cannot infer:
-
-- atom/molecule/conformer axes;
-- coordinate or energy units;
-- descriptor definitions;
-- train/test compound grouping;
-- assay censoring or detection limits; or
-- chemical identity from field names.
-
-HDF5 object names/attributes are redacted, external/soft links are not followed,
-and dataset values are not read. NumPy object arrays are rejected. Pickled
-models or RDKit objects are never deserialized.
-
-## Chemistry EDA rigor
-
-1. Define the independent unit: compound, batch, conformer, frame, calculation,
-   assay plate, specimen, or replicate.
-2. Preserve raw structures and measured values; record standardization as a
-   derived transformation.
-3. Create a data dictionary with units, assay endpoints, bounds, censoring,
-   LOD/LOQ, qualifiers, and provenance.
-4. Distinguish missing, failed, inactive, below detection, above quantitation,
-   and structurally invalid records.
-5. Split related analogues, scaffolds, batches, time, sites, or subjects before
-   learned preprocessing to prevent leakage. Random row splits can be
-   misleading.
-6. Compare robust/classical summaries and investigate outliers against
-   measurement and structure; do not delete automatically.
-7. Treat transformations (for example log concentration) as scientifically
-   defined and retain units/inverse interpretation.
-8. Label descriptor/property screening as exploratory and define multiplicity
-   control for inferential follow-up.
-9. Do not infer binding, efficacy, toxicity, mechanism, or causal effects from
-   EDA alone.
-
-## Recommended reference-only tooling
-
-Pin and validate tooling per project rather than treating this list as bundled
-support:
-
-- Gemmi or Biopython for PDBx/mmCIF/PDB;
-- RDKit or Open Babel for Molfile/SDF/SMILES;
-- ASE for XYZ and computational structures;
-- MDAnalysis or MDTraj for topology/trajectory pairs; and
-- cclib for supported quantum-chemistry outputs.
-
-Check each parser's current format table and release notes. Never pass untrusted
-property text to shell commands or dynamic evaluation.
-
-## Authoritative sources
-
-All links accessed 2026-07-23.
-
-- wwPDB, [File Formats and the PDB](https://www.wwpdb.org/documentation/file-formats-and-the-pdb)
-  (PDBx/mmCIF is the official archive/working format; legacy PDB format 3.30
-  where representable).
-- wwPDB, [PDBx/mmCIF Dictionary Resources](https://mmcif.wwpdb.org/) and
-  [current user guide](https://mmcif.wwpdb.org/docs/user-guide/guide.html).
-- wwPDB, [legacy PDB format 3.30](https://www.wwpdb.org/documentation/file-format-content/format33/v3.3.html).
-- IUCr, [CIF format specifications](https://www.iucr.org/resources/cif/spec)
-  (links to CIF 1.1 and 2.0 syntax).
-- RDKit, [current file parsing API](https://www.rdkit.org/docs/GettingStartedInPython.html#reading-and-writing-molecules).
-- MDAnalysis, [supported topology and trajectory formats](https://userguide.mdanalysis.org/stable/formats/index.html).
-- cclib, [supported programs and data](https://cclib.github.io/data.html).
-- NIST/SEMATECH, [Exploratory Data Analysis](https://www.itl.nist.gov/div898/handbook/eda/eda.htm).
-- scikit-learn, [data leakage guidance](https://scikit-learn.org/stable/common_pitfalls.html).

package/skills/exploratory-data-analysis/references/general_scientific_formats.md

@@ -1,259 +0,0 @@
-# General Scientific Formats and EDA Rigor
-
-**Reviewed:** 2026-07-23
-**Scope:** Exact capabilities of the bundled scripts plus conservative,
-documented workflows for common tabular and array containers.
-
-## Capability boundary
-
-| Format | Bundled executable inspection | Depth |
-|---|---|---|
-| `.csv`, `.tsv` | Yes, Python standard library | Bounded UTF-8 rectangular scan; schema, missingness, aggregate statistics, duplicate hashes, group/split leakage, and sensitivity |
-| `.json` | Yes, Python standard library | Bounded strict whole-document parse; structure and type counts only |
-| `.npy` | Optional, `numpy==2.5.1` | Header/shape/dtype plus bounded numeric sample; `allow_pickle=False` |
-| `.npz` | Optional, `numpy==2.5.1` | ZIP member/size/ratio preflight, then bounded per-array inspection; `allow_pickle=False` |
-| `.h5`, `.hdf5` | Optional, `h5py==3.16.0` | Bounded hierarchy and dataset metadata; payloads, attributes, soft links, external links, and external storage are not read |
-| `.parquet`, `.feather` | No | Reference-only pandas/Polars/Arrow workflow |
-| `.xlsx`, `.xls` | No | Reference-only workbook review; formulas, links, hidden content, and macros require separate handling |
-| `.zarr`, `.nc`, `.mat`, `.fits` | No | Reference-only domain tooling |
-| Pickle/joblib/dill | **Never** | Deserialization is outside this skill's security boundary |
-
-“Bundled executable” means a bounded inspection exists; it does not mean
-complete-file semantic validation. Unknown suffixes fail closed. Compressed
-generic archives are not unpacked.
-
-## Safe local-file contract
-
-All bundled CLIs:
-
-1. accept only regular local files inside an explicit `--root`;
-2. reject URLs, `..` traversal, home expansion, symlinks, multiply linked
-   inputs, and special files;
-3. enforce byte, row, field, column, member, object, and report limits;
-4. use the registered suffix and, where unambiguous, verify a magic signature;
-5. never use generic binary/text guessing as a fallback;
-6. emit aggregate statistics and tokenized identifiers by default, never rows;
-7. treat labels, headers, metadata, and file text as untrusted data, not
-   instructions; and
-8. write private (`0600`) outputs atomically and refuse overwrite unless
-   `--force` is explicit.
-
-Hashes/tokens are deterministic pseudonyms, not anonymization. A file hash or a
-low-cardinality value token can still be linkable.
-
-## CSV and TSV
-
-### Bundled approach
-
-`tabular_profile.py`, `missingness_leakage_audit.py`, and
-`distribution_sensitivity.py` use Python's `csv` module with:
-
-- UTF-8/UTF-8-with-BOM decoding and strict errors;
-- a fixed delimiter selected from `.csv` or `.tsv`, not sniffed;
-- `strict=True`, a bounded `csv.field_size_limit`, fixed maximum columns, and
-  rectangular-row enforcement;
-- an explicit missing-code policy (empty/whitespace only unless the user adds
-  `--missing-token`);
-- streaming Welford moments and deterministic bounded samples; and
-- no row or raw categorical-value output.
-
-Delimiter, decimal convention, thousands separators, encodings, comment
-syntax, and missing codes are part of the data dictionary. Do not silently
-guess them.
-
-### pandas 3.0.5 (documented alternate backend)
-
-PyPI published `pandas==3.0.5` on 2026-07-22; it supersedes the yanked 3.0.4.
-When pandas is appropriate, preserve the same outer path/size checks and use
-bounded selections:
-
-```python
-import pandas as pd
-
-frame = pd.read_csv(
-    local_path,
-    nrows=100_000,
-    usecols=approved_columns,
-    dtype=declared_types,
-    na_values=declared_missing_codes,
-    keep_default_na=False,
-    on_bad_lines="error",
-)
-```
-
-`nrows` and `usecols` reduce work, but do not replace file-size, field-size, or
-privacy controls. Keep parsing errors visible. Do not use `on_bad_lines="skip"`
-for EDA because it changes the analyzed population.
-
-### Polars 1.43.0 (documented alternate backend)
-
-PyPI published `polars==1.43.0` on 2026-07-21. Current `polars.read_csv`
-supports `columns`, `schema`, `schema_overrides`, `null_values`,
-`infer_schema_length`, and `n_rows`. Its docs note that:
-
-- malformed non-RFC-4180 data may have undefined behavior;
-- `ignore_errors=False` is the safe default;
-- `infer_schema_length=None` scans the full data into memory; and
-- with multithreaded parsing, `n_rows` is not guaranteed as a strict upper
-  bound.
-
-Prevalidate a local path; do not pass URLs or rely on optional `fsspec`. For
-strict bounded EDA, the bundled standard-library scanner is the reference
-implementation.
-
-## Strict JSON
-
-Python's current `json` documentation warns that malicious JSON can consume
-substantial CPU and memory and recommends limiting input size. It also
-documents that the default decoder accepts `NaN`/`Infinity` and silently keeps
-the last duplicate object key.
-
-The bundled inspector therefore:
-
-- caps the file at 16 MiB for parsing;
-- requires UTF-8;
-- rejects duplicate keys and non-finite constants;
-- catches recursion/resource errors;
-- traverses at most 100,000 nodes; and
-- emits only root type, depth, type counts, collection sizes, and tokenized
-  top-level field identifiers.
-
-JSON Lines/NDJSON is not registered. Rename-and-guess is not allowed.
-
-## NumPy NPY and NPZ
-
-NumPy's NPY specification stores shape and dtype in a header. NPZ is a ZIP
-archive whose members are NPY files. Object arrays can contain pickled Python
-objects.
-
-The bundled inspector always uses:
-
-```python
-array = np.load(
-    local_path,
-    mmap_mode="r",
-    allow_pickle=False,
-    max_header_size=10_000,
-)
-```
-
-For NPZ it first rejects:
-
-- non-NPY members, directories, traversal paths, encryption, and duplicate or
-  excessive members;
-- declared uncompressed content above 128 MiB; and
-- a per-member compression ratio above 100.
-
-It then loads one array at a time with `allow_pickle=False`. Numeric summaries
-use at most 4,096 deterministic sample elements. Structured dtype field names
-are identifiers and are tokenized by default. Object dtype is rejected; there
-is no `allow_pickle` override.
-
-Memory mapping reduces array payload reads but does not make malformed headers
-or huge shapes harmless. The outer byte and header limits remain mandatory.
-
-## HDF5 and h5py
-
-HDF5 is a container, not a semantic schema. Generic HDF5 inspection does not
-validate AnnData/H5AD, Loom, Imaris, mzMLb, or a laboratory's custom layout.
-
-h5py documents hard, soft, and external links. Dereferencing an external link
-opens another file. The bundled inspector uses `getlink=True` to classify
-links and never follows soft or external links. It:
-
-- reports at most 1,000 objects and 16 group levels;
-- deduplicates hard-link aliases;
-- reports shapes, dtype classes, chunking, compression presence, virtual/external
-  storage flags, and attribute counts;
-- does not read dataset payloads or attribute values;
-- does not call array conversion, user-defined callbacks, or dynamic
-  evaluation; and
-- does not invoke HDF5 filter plugins to decode data.
-
-Do not copy external-link filenames, object names, or attributes into reports.
-Do not set or trust `HDF5_PLUGIN_PATH` for untrusted files.
-
-## Reference-only formats
-
-### Parquet and Feather
-
-Use a pinned Arrow/pandas/Polars environment after local path validation.
-Inspect schema and row-group metadata first, select approved columns, and bound
-rows. The bundled scripts do not parse these formats, so they are not part of
-automated support.
-
-### Excel
-
-Spreadsheets can contain formulas, external links, hidden sheets, names,
-comments, and macros. Never enable macros, formula evaluation, or linked-data
-refresh. Export a values-only review copy to CSV/TSV after a human validates
-sheet choice, units, formulas, and merged/hidden regions. Preserve the original.
-
-### Zarr and directory stores
-
-Zarr/OME-Zarr are directory or object-store layouts rather than single regular
-files. The local-file CLIs reject directories. Use a separately sandboxed,
-version-aware Zarr workflow with explicit store and codec allowlists.
-
-## Statistical EDA contract
-
-1. Preserve the raw file and create a data dictionary with units and provenance.
-2. Identify observational units, replicates, grouping, pairing, clustering,
-   batches, sites, and time order before pooling.
-3. Preserve missingness and censoring indicators. Do not automatically impute,
-   substitute LOD/2, or treat non-detects as zero.
-4. Compare classical and robust summaries. Outlier flags trigger measurement
-   review and sensitivity analysis, not automatic deletion.
-5. Record transformation formulas and scientific rationale; fit any learned
-   parameter on training data only and retain raw-scale results.
-6. Split subjects/groups/time before fitting imputers, scalers, feature
-   selection, PCA, or other preprocessing.
-7. Label post hoc patterns as exploratory. Define the hypothesis family and
-   FWER/FDR plan before confirmatory testing.
-8. Report effect sizes, uncertainty, assumptions, limitations, exact software
-   versions, commands, deterministic rules/seeds, and derived artifact hashes.
-9. Do not make causal claims from descriptive associations.
-
-## Pinned optional snapshot
-
-Verified from PyPI on 2026-07-23:
-
-```bash
-uv pip install \
-  "numpy==2.5.1" \
-  "pandas==3.0.5" \
-  "polars==1.43.0" \
-  "h5py==3.16.0"
-```
-
-NumPy 2.5.1 requires Python 3.12+. These are direct-package snapshots, not a
-transitive lock; record a lockfile for a real analysis.
-
-## Authoritative sources
-
-All links accessed 2026-07-23.
-
-- Python 3.14, [`csv` — CSV File Reading and Writing](https://docs.python.org/3/library/csv.html).
-- Python 3.14, [`json` — JSON encoder and decoder](https://docs.python.org/3/library/json.html).
-- NumPy 2.5, [input/output reference](https://numpy.org/doc/stable/reference/routines.io.html),
-  [`numpy.load`](https://numpy.org/doc/stable/reference/generated/numpy.load.html),
-  [NPY/NPZ format](https://numpy.org/doc/stable/reference/generated/numpy.lib.format.html),
-  and [security guidance](https://numpy.org/doc/stable/reference/security.html).
-- pandas 3.0, [I/O tools](https://pandas.pydata.org/docs/user_guide/io.html);
-  [PyPI 3.0.5](https://pypi.org/project/pandas/), released 2026-07-22.
-- Polars 1.43, [`polars.read_csv`](https://docs.pola.rs/api/python/stable/reference/api/polars.read_csv.html);
-  [PyPI 1.43.0](https://pypi.org/project/polars/), released 2026-07-21.
-- h5py 3.16, [groups and links](https://docs.h5py.org/en/stable/high/group.html);
-  [PyPI 3.16.0](https://pypi.org/project/h5py/), released 2026-03-06.
-- NIST/SEMATECH, [Exploratory Data Analysis](https://www.itl.nist.gov/div898/handbook/eda/eda.htm)
-  and [chapter references](https://www.itl.nist.gov/div898/handbook/eda/section4/eda43.htm).
-- Box and Cox (1964), [“An Analysis of Transformations”](https://doi.org/10.1111/j.2517-6161.1964.tb00553.x).
-- FDA/ICH E9(R1), [Estimands and Sensitivity Analysis](https://www.fda.gov/regulatory-information/search-fda-guidance-documents/e9r1-statistical-principles-clinical-trials-addendum-estimands-and-sensitivity-analysis-clinical),
-  final guidance May 2021.
-- US EPA, [Detection Limits Best Practices Guide](https://www.epa.gov/system/files/documents/2025-09/wqxdetectionlimitsbestpracticesguide_final.pdf),
-  dated August 2025.
-- scikit-learn, [common pitfalls and data leakage](https://scikit-learn.org/stable/common_pitfalls.html).
-- Benjamini and Hochberg (1995), [false discovery rate](https://academic.oup.com/jrsssb/article/57/1/289/7035855).
-- Wasserstein, Schirm, and Lazar (2019), [Moving to a World Beyond “p < 0.05”](https://doi.org/10.1080/00031305.2019.1583913).
-- National Academies (2019), [*Reproducibility and Replicability in Science*](https://doi.org/10.17226/25303).
-- Wilkinson et al. (2016), [FAIR Guiding Principles](https://doi.org/10.1038/sdata.2016.18).