@pikaa-ai/pikaa 0.3.23 → 0.3.24

package/skills/pydicom/SKILL.md

@@ -1,381 +0,0 @@
----
-name: pydicom
-description: Use pydicom to read, inspect, write, transform, and safely preflight local DICOM datasets and pixel data. Applies to DICOM metadata, transfer syntaxes, compression plugins, frames, private elements, JSON, and bounded de-identification review.
-license: MIT
-compatibility: Python 3.10+ with pydicom 3.0.2; optional pinned NumPy, Pillow, and pixel plugins. Helper CLIs are local-only and require authorized data.
-metadata:
-  version: "1.1"
-  skill-author: "K-Dense Inc."
-  last-reviewed: "2026-07-23"
----
-
-# pydicom
-
-Use pydicom for DICOM dataset I/O and pixel processing. Version 3.0.2 is the
-current stable release reviewed here. It fixes CVE-2026-32711, a crafted
-DICOMDIR path-traversal issue. pydicom 3.0.2 declares Python `>=3.10`; its
-bundled DICOM dictionary is 2024c, while the live DICOM Standard may be newer.
-
-## Mandatory safety boundary
-
-- Work only with local data that the user is authorized to access.
-- DICOM metadata, file names, private elements, overlays, structured content,
-  and pixels may contain protected health information (PHI).
-- Never print `Dataset`, export full metadata/JSON, or log element values by
-  default. Use a documented allowlist and aggregate output.
-- pydicom is a general DICOM framework, not a diagnostic viewer. Pixel output,
-  validation, conversion, and plugin availability are not diagnostic claims.
-- De-identification is profile-, purpose-, recipient-, jurisdiction-, and
-  threat-context-specific. It requires privacy/DICOM expert verification.
-- Never claim that a tag-removal script is DICOM PS3.15, HIPAA, GDPR, or other
-  compliance. Preserve originals and audit derived outputs.
-- Treat deterministic pseudonymization keys and UID maps as re-identification
-  secrets: use least privilege and encrypted/managed secret storage, never
-  commit, sync, log, or share them with derivatives, and define backup,
-  rotation, revocation, and destruction procedures. A leaked key invalidates
-  the intended separation; rotation also changes deterministic mappings.
-- Set explicit input-file, file-count, frame-count, decoded-byte, and output
-  limits before parsing untrusted or unusually large datasets.
-
-## Installation
-
-Create or activate an isolated environment, then install the exact reviewed
-release:
-
-```bash
-uv pip install "pydicom==3.0.2"
-```
-
-Uncompressed pixel arrays and image rendering:
-
-```bash
-uv pip install "pydicom==3.0.2" "numpy==2.5.1" "Pillow==12.3.0"
-```
-
-Install only the transfer-syntax plugins required by the deployment:
-
-```bash
-# JPEG/JPEG-LS, JPEG 2000/HTJ2K, and faster RLE through pylibjpeg
-uv pip install "numpy==2.5.1" "pylibjpeg==2.1.0" \
-  "pylibjpeg-libjpeg==2.4.0" "pylibjpeg-openjpeg==2.5.0" \
-  "pylibjpeg-rle==2.2.0"
-
-# JPEG-LS encoder/decoder
-uv pip install "numpy==2.5.1" "pyjpegls==1.5.1"
-
-# Alternative decoder with platform-specific wheels
-uv pip install "python-gdcm==3.2.6"
-```
-
-Plugin licenses and wheels differ by package/platform; review them before
-deployment. Pillow has documented decoding limitations and pydicom cautions
-that plugin output must be independently checked.
-
-Native codec wheels widen the supply-chain and memory-safety boundary. For a
-controlled deployment, resolve these exact pins on a trusted build host, lock
-and verify wheel hashes/provenance, mirror approved artifacts internally, scan
-them, and install with hash enforcement rather than resolving from the public
-index at runtime.
-
-## Choose the workflow
-
-1. Need an aggregate overview: run `scripts/extract_metadata.py`.
-2. Need bounded technical checks: run `scripts/dicom_inventory.py`.
-3. Need codec deployment preflight: run
-   `scripts/transfer_syntax_inspector.py`.
-4. Need frame/memory planning: run `scripts/pixel_frame_planner.py`.
-5. Need one non-diagnostic rendered frame: run
-   `scripts/dicom_to_image.py`.
-6. Need a pseudonymized derivative: read the de-identification section, create
-   a site-reviewed action profile, then run `scripts/anonymize_dicom.py` and
-   `scripts/deidentification_audit.py`.
-7. Need to check a sensitive UID map: run
-   `scripts/uid_mapping_validator.py`.
-
-## Read datasets safely
-
-`dcmread()` returns a `FileDataset`, a `Dataset` subclass with File Format
-state such as `file_meta`, preamble, and original encoding.
-
-```python
-from pathlib import Path
-import pydicom
-
-path = Path("authorized/input.dcm")
-ds = pydicom.dcmread(
-    path,
-    stop_before_pixels=True,
-    specific_tags=[
-        "SOPClassUID",
-        "Modality",
-        "Rows",
-        "Columns",
-        "NumberOfFrames",
-    ],
-)
-
-technical = {
-    "sop_class": ds.get("SOPClassUID"),
-    "modality": ds.get("Modality"),
-    "rows": ds.get("Rows"),
-    "columns": ds.get("Columns"),
-}
-```
-
-Use:
-
-- `stop_before_pixels=True` for metadata-only work.
-- `specific_tags=[...]` for a minimum allowlist.
-- `defer_size="1 MiB"` when a later write must preserve large values.
-- `force=False` (default). `force=True` only bypasses the File Format header
-  check; it does not prove the bytes are valid DICOM.
-
-Do not call `print(ds)`, `repr(ds)`, or iterate values into logs on clinical
-data.
-
-## Dataset, DataElement, and sequences
-
-Access standard elements by keyword and check for absence:
-
-```python
-modality = ds.get("Modality", "UNSPECIFIED")
-if "ReferencedImageSequence" in ds:
-    for item in ds.ReferencedImageSequence:
-        referenced_class = item.get("ReferencedSOPClassUID")
-```
-
-Tag access, such as `ds[0x0010, 0x0010]`, returns a `DataElement`; its `.value`
-is separate. `Sequence` behaves like a list of nested `Dataset` items. Privacy
-actions must recurse through every sequence item, not only the top level.
-
-When creating a file, use `FileMetaDataset` for group `0002`, keep dataset and
-file-meta SOP UIDs consistent, set a Transfer Syntax UID, and write in enforced
-File Format:
-
-```python
-from pydicom import dcmwrite
-from pydicom.dataset import FileDataset, FileMetaDataset
-from pydicom.uid import CTImageStorage, ExplicitVRLittleEndian, generate_uid
-
-meta = FileMetaDataset()
-meta.MediaStorageSOPClassUID = CTImageStorage
-meta.MediaStorageSOPInstanceUID = generate_uid()
-meta.TransferSyntaxUID = ExplicitVRLittleEndian
-
-ds = FileDataset(None, {}, file_meta=meta, preamble=b"\0" * 128)
-ds.SOPClassUID = meta.MediaStorageSOPClassUID
-ds.SOPInstanceUID = meta.MediaStorageSOPInstanceUID
-# Add all attributes required by the selected IOD before writing.
-dcmwrite("new.dcm", ds, enforce_file_format=True, overwrite=False)
-```
-
-`write_like_original` is deprecated in pydicom 3.0; use
-`enforce_file_format`. A successful write is not full PS3.3 IOD conformance.
-
-## UIDs and transfer syntax
-
-The File Meta Information Transfer Syntax UID controls dataset encoding and
-pixel compression:
-
-```python
-ts = ds.file_meta.TransferSyntaxUID
-summary = {
-    "uid": str(ts),
-    "name": ts.name,
-    "compressed": ts.is_compressed,
-    "implicit_vr": ts.is_implicit_VR,
-    "little_endian": ts.is_little_endian,
-}
-```
-
-pydicom 3.0 chooses write encoding from the Transfer Syntax UID before legacy
-dataset flags. Do not replace structural UIDs (Transfer Syntax, SOP Class, or
-coding-scheme UIDs) during pseudonymization. Instance/reference UID replacement
-must be one-to-one and consistent across the complete declared scope.
-
-Read [references/transfer_syntaxes.md](references/transfer_syntaxes.md) before
-compression, decompression, or encapsulation.
-
-## Pixel data and frames
-
-The stable `pydicom.pixels` API supports path-based, frame-specific decoding:
-
-```python
-from pydicom.pixels import pixel_array
-
-# Reads only the selected frame where the source permits it.
-frame = pixel_array("authorized/image.dcm", index=0, raw=False)
-```
-
-Shape semantics:
-
-- grayscale single frame: `(rows, columns)`
-- grayscale multi-frame: `(frames, rows, columns)`
-- color single frame: `(rows, columns, samples)`
-- color multi-frame: `(frames, rows, columns, samples)`
-
-`raw=False` converts YCbCr pixel data to RGB when possible; `raw=True` retains
-the decoded color space after mandatory minimal processing. Use
-`iter_pixels(path, indices=[...])` for bounded multi-frame iteration.
-
-For grayscale display, apply transforms in this order:
-
-```python
-from pydicom.pixels import apply_modality_lut, apply_voi_lut
-
-modality_values = apply_modality_lut(frame, ds)
-display_values = apply_voi_lut(modality_values, ds, index=0)
-```
-
-Modality LUT/rescale and VOI/windowing change display/value semantics.
-MONOCHROME1 may require presentation inversion. Palette Color requires
-`apply_color_lut()`. Presentation states and ICC behavior may require a
-validated viewer. Never use per-frame min/max normalization for quantitative
-analysis.
-
-## Compression, decompression, and encapsulation
-
-- Accessing `pixel_array` decodes as needed but does not change the dataset.
-- `Dataset.decompress()` changes Pixel Data in place, sets Explicit VR Little
-  Endian, updates image metadata, and generates a new SOP Instance UID by
-  default.
-- `Dataset.compress(uid)` changes Pixel Data and Transfer Syntax in place and
-  generates a new SOP Instance UID by default.
-- pydicom 3.0 built-in/found encoders cover RLE Lossless, JPEG-LS, and JPEG
-  2000 combinations documented in the stable plugin matrix.
-- Each compressed frame is separately encoded and then encapsulated. Use
-  `encapsulate()` or `encapsulate_extended()` for externally encoded frames.
-- Read frames with current `pydicom.encaps.generate_frames()` or `get_frame()`;
-  legacy encapsulation generator names are deprecated for pydicom 4.
-
-Always inspect capabilities first, limit decoded bytes/frames, and verify pixel
-correctness independently. Lossy compression acceptability is outside pydicom
-and the DICOM encoding specification.
-
-## DICOM JSON and private elements
-
-`Dataset.to_json()`, `to_json_dict()`, and `Dataset.from_json()` implement the
-DICOM JSON Model, but pydicom documents JSON support as beta. Full JSON may
-inline binary data and expose every identifier and pixel payload. Do not emit
-it as a metadata report. A `BulkDataURI` handler introduces separate storage,
-authorization, and retrieval obligations.
-
-Private elements are not standardized and may contain PHI:
-
-```python
-# Recursive removal, but not sufficient de-identification by itself.
-ds.remove_private_tags()
-```
-
-Retain private elements only under an explicit reviewed safe-private policy.
-Read [references/common_tags.md](references/common_tags.md) for tag access,
-privacy classes, and standard pointers.
-
-## De-identification workflow
-
-DICOM PS3.15 Annex E explicitly states that confidentiality profiles do not
-guarantee removal of all identifying information and do not replace a complete
-de-identification process.
-
-1. Define purpose, recipients, linkage needs, regulations, threat model, and
-   acceptable re-identification risk.
-2. Select the Basic Application Level Confidentiality Profile and needed
-   options (pixel, recognizable visual features, graphics, structured content,
-   descriptors, temporal information, patient characteristics, devices,
-   institutions, UIDs, and safe private data).
-3. Preserve source objects unchanged in controlled storage.
-4. Apply every action recursively, including nested sequences.
-5. Replace instance/reference UIDs consistently across the complete scope;
-   preserve structural UIDs.
-6. Decide date/time handling explicitly. A fixed shift can preserve intervals
-   but partial dates, time zones, standalone times, leap days, longitudinal
-   linkage, and external events require reviewed policy.
-7. Inspect pixels, overlays, graphics, structured content, and recognizable
-   visual features. Do not infer clean pixels from missing metadata or set
-   `BurnedInAnnotation=NO` without verification.
-8. Rebuild File Meta Information and preamble to prevent leakage.
-9. Run technical validation and a de-identification audit, then perform expert
-   verification and documented risk review.
-
-The bundled script intentionally sets `PatientIdentityRemoved` to `NO` because
-it cannot establish successful de-identification.
-
-## Helper CLIs
-
-All `--help` paths are dependency-free. The tools perform no network access and
-emit no DICOM values beyond narrow technical allowlists.
-
-Bundled content consists of the two linked references, the documented helper
-scripts, and synthetic tests. The pydicom runtime dependency is installed from
-the pinned PyPI release.
-
-```bash
-# Redacted aggregate metadata
-python scripts/extract_metadata.py authorized/ --recursive
-
-# Metadata-only technical inventory
-python scripts/dicom_inventory.py authorized/ --recursive
-
-# Installed codec/plugin capabilities
-python scripts/transfer_syntax_inspector.py --input authorized/image.dcm
-
-# Frame shape, byte, and transform plan
-python scripts/pixel_frame_planner.py authorized/image.dcm --frames 0,2-4
-
-# One non-diagnostic frame
-python scripts/dicom_to_image.py authorized/image.dcm frame.png \
-  --acknowledge-pixel-phi
-
-# Create a secret key, then a scoped pseudonymized derivative plus audit
-python scripts/anonymize_dicom.py --generate-uid-key project.key
-python scripts/anonymize_dicom.py authorized/in.dcm derived/out.dcm \
-  --uid-key-file project.key --uid-scope export-v1 \
-  --audit-report derived/out.audit.json
-
-# Audit candidate metadata; no pixel decompression
-python scripts/deidentification_audit.py derived/out.dcm
-
-# Validate an explicitly requested sensitive UID mapping
-python scripts/uid_mapping_validator.py derived/uid-map.json \
-  --uid-key-file project.key --uid-scope export-v1
-```
-
-The generated raw key file is a controlled-local convenience and is created
-with owner-only permissions. For production, materialize key bytes from an
-approved secret manager into a locked ephemeral file, restrict access to the
-de-identification service, and securely remove it afterward. Store any optional
-UID map separately from derivatives; it directly links original and replacement
-identifiers.
-
-## pydicom 3.0 migration notes
-
-- `read_file()` and `write_file()` were removed; use `dcmread()` and
-  `dcmwrite()`.
-- `write_like_original` is deprecated; use `enforce_file_format`.
-- `pydicom.pixel_data_handlers` is deprecated for removal in v4; use
-  `pydicom.pixels`.
-- `Dataset.pixel_array` uses the new pixels backend by default and converts
-  YCbCr to RGB when possible.
-- `JPEGLossless` now means UID `1.2.840.10008.1.2.4.57`;
-  `JPEGLosslessSV1` is `.70`.
-- `Dataset.is_little_endian` and `is_implicit_VR` are deprecated for v4.
-
-## Sources (verified 2026-07-23)
-
-- [pydicom 3.0.2 on PyPI](https://pypi.org/project/pydicom/) — released
-  2026-03-19; Python `>=3.10`.
-- [pydicom releases](https://github.com/pydicom/pydicom/releases) — 3.0.2 and
-  CVE-2026-32711 details.
-- [Stable release notes](https://pydicom.github.io/pydicom/stable/release_notes/index.html)
-- [Stable installation guide](https://pydicom.github.io/pydicom/stable/tutorials/installation.html)
-- [Dataset basics](https://pydicom.github.io/pydicom/stable/tutorials/dataset_basics.html)
-- [Stable pixel tutorial](https://pydicom.github.io/pydicom/stable/tutorials/pixel_data/introduction.html)
-- [Stable pixel plugins](https://pydicom.github.io/pydicom/stable/guides/user/image_data_handlers.html)
-- [Stable compression tutorial](https://pydicom.github.io/pydicom/stable/tutorials/pixel_data/compressing.html)
-- [Stable DICOM JSON tutorial](https://pydicom.github.io/pydicom/stable/tutorials/dicom_json.html)
-- [Stable private-element guide](https://pydicom.github.io/pydicom/stable/guides/user/private_data_elements.html)
-- [Current DICOM Standard](https://www.dicomstandard.org/current)
-- [DICOM PS3.3](https://dicom.nema.org/medical/dicom/current/output/chtml/part03/PS3.3.html),
-  [PS3.5](https://dicom.nema.org/medical/dicom/current/output/chtml/part05/PS3.5.html),
-  [PS3.6](https://dicom.nema.org/medical/dicom/current/output/chtml/part06/PS3.6.html),
-  and [PS3.15](https://dicom.nema.org/medical/dicom/current/output/html/part15.html)

package/skills/pyhealth/SKILL.md

@@ -1,124 +0,0 @@
----
-name: pyhealth
-description: Build clinical/healthcare deep-learning pipelines with PyHealth — loading EHR/signal/imaging datasets (MIMIC-III/IV, eICU, OMOP, SleepEDF, ChestXray14, EHRShot), defining tasks (mortality, readmission, length-of-stay, drug recommendation, sleep staging, ICD coding, EEG events), instantiating models (Transformer, RETAIN, GAMENet, SafeDrug, MICRON, StageNet, AdaCare, CNN/RNN/MLP), training with the PyHealth Trainer, computing clinical metrics, and using medical code utilities (ICD/ATC/NDC/RxNorm lookup and cross-mapping). Use this skill whenever the user mentions PyHealth, MIMIC, eICU, OMOP, EHR modeling, clinical prediction, drug recommendation, sleep staging, medical code mapping, ICD/ATC codes, or any healthcare ML pipeline that fits the dataset → task → model → trainer → metrics pattern, even if "PyHealth" isn't named explicitly.
-metadata:
-  version: "1.0"
-  skill-author: K-Dense Inc.
----
-
-# PyHealth
-
-PyHealth (https://pyhealth.dev/) is a Python toolkit for clinical deep learning. It provides a unified, modular pipeline across electronic health records (EHR), physiological signals, and medical imaging.
-
-The library is built around a **5-stage pipeline** — `Dataset → Task → Model → Trainer → Metrics` — where each stage is replaceable and the interfaces between stages are stable. Code that follows this pipeline shape composes well; code that bypasses it usually fights the library.
-
-## When to use this skill
-
-Use this skill whenever the user is doing clinical/healthcare ML and any of the following are true:
-
-- They mention PyHealth, MIMIC-III/IV, eICU, OMOP-CDM, EHRShot, SleepEDF, SHHS, ISRUC, COVID19-CXR, ChestX-ray14, TUEV/TUAB.
-- They want to predict mortality, readmission, length of stay, drug recommendations, sleep stages, ICD codes, EEG events, or de-identification.
-- They need to look up or cross-map medical codes (ICD-9-CM, ICD-10-CM, ATC, NDC, RxNorm, CCS).
-- They have EHR-shaped data and want to train a clinical model without writing the plumbing themselves.
-
-PyHealth is the right tool when the workflow fits its 5 stages. If the user just wants generic PyTorch on tabular data, this skill is not necessary.
-
-## Installation (uv)
-
-PyHealth 2.0 requires Python ≥ 3.12, < 3.14. Use `uv` for environment management — it's faster and reproducible.
-
-```bash
-# Create a project with the right Python
-uv init my-pyhealth-project
-cd my-pyhealth-project
-uv python pin 3.12
-
-# Add PyHealth (this also pulls in PyTorch and friends)
-uv add pyhealth
-
-# Run scripts inside the env
-uv run python train.py
-```
-
-For a one-off script without a project, use `uv run --with pyhealth python script.py`. For the legacy 1.x line (Python 3.9+), `uv add pyhealth==1.16`. Detailed install notes, MIMIC access, and GPU/CPU device tips are in `references/installation.md`.
-
-## The 5-stage pipeline
-
-A complete pipeline is typically <20 lines. This is the canonical shape — start here and modify pieces:
-
-```python
-from pyhealth.datasets import MIMIC3Dataset, split_by_patient, get_dataloader
-from pyhealth.tasks import MortalityPredictionMIMIC3
-from pyhealth.models import Transformer
-from pyhealth.trainer import Trainer
-from pyhealth.metrics.binary import binary_metrics_fn
-
-# 1. Dataset — raw patient registry
-base = MIMIC3Dataset(
-    root="https://storage.googleapis.com/pyhealth/Synthetic_MIMIC-III/",
-    tables=["DIAGNOSES_ICD", "PROCEDURES_ICD", "PRESCRIPTIONS"],
-)
-
-# 2. Task — converts patients into supervised samples
-samples = base.set_task(MortalityPredictionMIMIC3())
-
-# 3. Split + DataLoaders (split by patient to avoid leakage)
-train_ds, val_ds, test_ds = split_by_patient(samples, [0.8, 0.1, 0.1])
-train_loader = get_dataloader(train_ds, batch_size=32, shuffle=True)
-val_loader   = get_dataloader(val_ds,   batch_size=32, shuffle=False)
-test_loader  = get_dataloader(test_ds,  batch_size=32, shuffle=False)
-
-# 4. Model — must be passed the SampleDataset, not the BaseDataset
-model = Transformer(dataset=samples)
-
-# 5. Train + evaluate
-trainer = Trainer(model=model)
-trainer.train(
-    train_dataloader=train_loader,
-    val_dataloader=val_loader,
-    epochs=50,
-    monitor="pr_auc",
-)
-
-y_true, y_prob, _ = trainer.inference(test_loader)
-print(binary_metrics_fn(y_true, y_prob, metrics=["pr_auc", "roc_auc"]))
-```
-
-A copy-pasteable starter is in `assets/starter_pipeline.py`.
-
-## Critical things to get right
-
-These are the mistakes that PyHealth code most commonly trips on. Internalize them before writing pipelines:
-
-1. **Models take a `SampleDataset`, not a `BaseDataset`.** `MIMIC3Dataset(...)` returns a `BaseDataset` (a queryable patient registry). Only after `.set_task(task)` do you get a `SampleDataset`, which is what models, splitters, and DataLoaders expect. If you pass `base` to a model, it will fail or behave wrong.
-
-2. **Always split by patient (or visit), not by sample.** Random sample-level splits leak information across train/test because the same patient can appear in both. Use `split_by_patient` for patient-level prediction, `split_by_visit` only when visits are independent.
-
-3. **Match the task to the dataset.** Tasks are dataset-specific: `MortalityPredictionMIMIC3` won't work on MIMIC-IV — use `MortalityPredictionMIMIC4` or `InHospitalMortalityMIMIC4`. The full mapping is in `references/tasks.md`.
-
-4. **Pick `monitor` to match the task type.** For binary classification use `"pr_auc"` or `"roc_auc"`. For multilabel (drug rec) use `"pr_auc_samples"` or `"jaccard_samples"`. For multiclass use `"accuracy"` or `"f1_macro"`. Wrong monitor → checkpoint selection saves the wrong epoch.
-
-5. **MIMIC-IV uses `ehr_root=`, not `root=`.** This is the one inconsistency in the dataset constructors.
-
-6. **For reproducible work, point `cache_dir=` somewhere persistent.** PyHealth caches the parsed dataset; without `cache_dir`, you re-parse every run.
-
-## How to use this skill
-
-PyHealth has a large API surface — there's no point loading it all at once. Read the reference file that matches the user's task:
-
-| If the user is asking about… | Read |
-|---|---|
-| Installing, env setup, MIMIC access, GPU | `references/installation.md` |
-| Which dataset class to use, loading patterns, splitting | `references/datasets.md` |
-| What prediction task to choose (mortality, readmission, drug rec, sleep…) | `references/tasks.md` |
-| Picking a model architecture, model-specific arguments | `references/models.md` |
-| Looking up or cross-mapping ICD/ATC/NDC/RxNorm/CCS codes, tokenizers | `references/medcode.md` |
-| End-to-end recipes for common scenarios | `references/examples.md` |
-
-For multi-step tasks (e.g., "build a drug recommendation pipeline on MIMIC-IV"), read `tasks.md` + `models.md` + `examples.md` together — they cross-reference each other.
-
-## A note on style
-
-Write minimal, idiomatic PyHealth. The library is opinionated; lean into its abstractions instead of reimplementing them in raw PyTorch. If you find yourself writing a custom training loop, ask whether `Trainer` would do the job — it almost always will, and it handles checkpointing, logging, and best-model selection for free.
-
-When the user has private MIMIC access, point them at the local CSV root; for demos and learning, the synthetic MIMIC-III bucket (`https://storage.googleapis.com/pyhealth/Synthetic_MIMIC-III/`) is fine and works without credentialing.