@pikaa-ai/pikaa 0.2.4 → 0.3.0

package/skills/scikit-survival/SKILL.md

@@ -0,0 +1,313 @@
+---
+name: scikit-survival
+description: Build, evaluate, and audit right-censored or competing-risk survival workflows with scikit-survival, including leakage-safe preprocessing, model selection, probability prediction, and censoring-aware metrics.
+license: MIT
+compatibility: Requires Python 3.11+, uv, and the pinned scikit-survival 0.28.0 stack for executable examples. Bundled CLIs are local and network-free by default.
+allowed-tools: Read Write Edit Bash
+metadata:
+  version: "1.1"
+  skill-author: K-Dense Inc.
+---
+
+# scikit-survival
+
+## Scope
+
+Use this skill for scikit-survival 0.28.0 workflows involving:
+
+- right-censored structured outcomes;
+- Cox PH, Coxnet, IPC ridge, survival trees, forests, boosting, and SVMs;
+- discrimination, prediction error, calibration-oriented checks, and time-dependent prediction;
+- nonparametric cumulative incidence with competing risks;
+- scikit-learn pipelines, nested model selection, and reproducible reports.
+
+scikit-survival primarily models right-censored outcomes. Its built-in competing-risk
+support is nonparametric cumulative incidence; it does not provide Fine-Gray regression.
+Do not present model output as clinical advice, causal evidence, or proof of clinical
+utility.
+
+## Current release and installation
+
+Verified 2026-07-23:
+
+- Latest stable: **scikit-survival 0.28.0**, released 2026-07-05.
+- Python: **3.11 or later**; PyPI wheels cover CPython 3.11-3.14 on Linux
+  x86-64, macOS x86-64/ARM64, and Windows x86-64.
+- Runtime bounds: NumPy >=2.0.0, pandas >=2.2.0, SciPy >=1.13.0,
+  scikit-learn >=1.9.0,<1.10, OSQP >=1.0.2, narwhals >=2.0.1.
+- 0.28 adds pandas/Polars estimator support through narwhals and removes
+  `criterion` from `GradientBoostingSurvivalAnalysis`.
+
+Create an isolated environment and install the tested snapshot:
+
+```bash
+uv venv --python 3.11
+source .venv/bin/activate
+uv pip install \
+  "scikit-survival==0.28.0" \
+  "scikit-learn==1.9.0" \
+  "numpy==2.4.6" \
+  "pandas==3.0.5" \
+  "scipy==1.17.1" \
+  "ecos==2.0.14" \
+  "osqp==1.1.3" \
+  "joblib==1.5.3" \
+  "numexpr==2.14.2" \
+  "narwhals==2.24.0"
+```
+
+Binary wheels are preferred. A source build requires a C/C++ compiler; OSQP may
+also require CMake. This skill is MIT-licensed; the upstream scikit-survival package
+is GPL-3.0-or-later, so review upstream licensing before redistribution.
+
+## Non-negotiable workflow
+
+1. **Define the estimand and event coding.** Decide whether the target is
+   all-event survival, cause-specific hazard, or cause-specific cumulative incidence.
+2. **Validate outcomes.** Standard estimators need a two-field structured array:
+   boolean event first, observed time second. Competing-risk CIF instead needs a
+   separate integer event vector: 0=censored, 1..K=causes.
+3. **Split before learned preprocessing.** Never fit imputers, encoders, scalers,
+   feature selectors, or alpha choices on all rows before splitting.
+4. **Fit preprocessing inside a pipeline.** Unknown categories and missingness must
+   be handled using training-fold state only.
+5. **Tune without reusing evaluation data.** Use nested CV when reporting
+   cross-validated tuned performance, or reserve a truly untouched final holdout.
+6. **Fit censoring distributions on training data.** IPCW concordance, dynamic AUC,
+   and Brier metrics receive `survival_train`, never a pooled train+test outcome.
+7. **Restrict evaluation times.** Use a strictly increasing grid inside test
+   follow-up and below the end of training support where the estimated censoring
+   survival remains positive.
+8. **Match predictions to metrics.** Concordance/dynamic AUC consume higher-is-riskier
+   scores. Brier metrics consume survival probabilities with shape
+   `(n_test, n_times)`, not risk scores or unevaluated step functions.
+9. **Handle competing causes explicitly.** Standard survival probabilities and CIFs
+   answer different questions. Never estimate event-specific probability with
+   `1 - Kaplan-Meier` while censoring competing events.
+10. **Report limits.** Separate discrimination, calibration, prediction error,
+    and cumulative incidence. None alone establishes decision or clinical utility.
+
+## Outcome construction
+
+```python
+from sksurv.util import Surv
+
+y = Surv.from_arrays(event=event_bool, time=observed_time)
+# Equivalent for pandas or Polars:
+y = Surv.from_dataframe("event", "time", frame)
+```
+
+The first field is boolean (`True`=event, `False`=right-censored); the second is
+floating-point time. Field names may vary, but field order and meaning may not.
+Use `references/data-handling.md` before loading custom or competing-risk data.
+
+## Leakage-safe pipeline
+
+```python
+from sklearn.compose import ColumnTransformer
+from sklearn.impute import SimpleImputer
+from sklearn.model_selection import train_test_split
+from sklearn.pipeline import make_pipeline
+from sklearn.preprocessing import OneHotEncoder, StandardScaler
+from sksurv.linear_model import CoxPHSurvivalAnalysis
+
+X_train, X_test, y_train, y_test = train_test_split(
+    X, y, test_size=0.25, stratify=y["event"], random_state=20260723
+)
+
+preprocess = ColumnTransformer(
+    [
+        ("num", make_pipeline(SimpleImputer(strategy="median"), StandardScaler()), numeric),
+        (
+            "cat",
+            make_pipeline(
+                SimpleImputer(strategy="most_frequent"),
+                OneHotEncoder(handle_unknown="ignore", drop="first", sparse_output=False),
+            ),
+            categorical,
+        ),
+    ],
+    sparse_threshold=0.0,
+)
+model = make_pipeline(preprocess, CoxPHSurvivalAnalysis(alpha=0.1, ties="efron"))
+model.fit(X_train, y_train)
+risk = model.predict(X_test)
+```
+
+The split precedes every learned transformation. For repeated or grouped records,
+use a group-aware split; for temporal deployment, use a time-respecting split.
+
+## Model choice
+
+- `CoxPHSurvivalAnalysis`: interpretable log-hazard coefficients under proportional
+  hazards; `alpha` is ridge shrinkage and `ties` is `"breslow"` or `"efron"`.
+- `CoxnetSurvivalAnalysis`: LASSO/elastic-net path for high-dimensional data.
+  `l1_ratio` is in `(0, 1]`; use `fit_baseline_model=True` before requesting
+  survival or cumulative-hazard functions.
+- `IPCRidge`: IPC-weighted ridge AFT model; prediction is on a time/log-time scale,
+  not a Cox risk score.
+- `RandomSurvivalForest` / `ExtraSurvivalTrees`: nonlinear survival and cumulative
+  hazard predictions; use permutation importance, not impurity importance.
+- `GradientBoostingSurvivalAnalysis`: tree boosting with `"coxph"`, `"squared"`,
+  or `"ipcwls"` loss. `criterion` was removed in 0.28.
+- `ComponentwiseGradientBoostingSurvivalAnalysis`: sparse linear componentwise
+  boosting.
+- `FastSurvivalSVM` / `FastKernelSurvivalSVM`: ranking or regression objectives.
+  Only `rank_ratio=1` directly returns higher-is-riskier scores; SVMs do not yield
+  survival probabilities for Brier metrics.
+
+Read the model-specific reference before interpreting coefficients or predictions:
+`references/cox-models.md`, `references/ensemble-models.md`, or
+`references/svm-models.md`.
+
+## Prediction and metric contracts
+
+```python
+import numpy as np
+from sksurv.metrics import (
+    brier_score,
+    concordance_index_ipcw,
+    cumulative_dynamic_auc,
+    integrated_brier_score,
+)
+
+risk = model.predict(X_test)  # (n_test,), higher means higher event risk
+uno_c = concordance_index_ipcw(y_train, y_test, risk, tau=times[-1])[0]
+auc_t, mean_auc = cumulative_dynamic_auc(y_train, y_test, risk, times)
+
+surv_fns = model.predict_survival_function(X_test)
+surv_prob = np.vstack([fn(times) for fn in surv_fns])  # (n_test, n_times)
+_, brier_t = brier_score(y_train, y_test, surv_prob, times)
+ibs = integrated_brier_score(y_train, y_test, surv_prob, times)
+```
+
+- Harrell C and Uno C measure rank discrimination, not calibration.
+- Cumulative/dynamic AUC measures discrimination at selected horizons and accepts
+  1D or time-dependent 2D risk scores; it rejects survival probabilities.
+- Brier score is censoring-weighted probability error and reflects both
+  discrimination and calibration. It is not a standalone calibration curve.
+- Calibration requires horizon-specific predicted-versus-observed checks on
+  independent data. scikit-survival 0.28 has no dedicated calibration-curve API.
+
+See `references/evaluation-metrics.md` for assumptions, primary literature, safe
+time-grid construction, and scorer wrappers.
+
+## Pipelines, metadata routing, and tuning
+
+Ordinary `Pipeline.fit(X, y)` needs no metadata-routing setup. Metric wrappers such
+as `as_concordance_index_ipcw_scorer` are estimator wrappers, not `scoring=`
+callables:
+
+```python
+from sklearn.model_selection import GridSearchCV
+from sksurv.metrics import as_concordance_index_ipcw_scorer
+
+wrapped = as_concordance_index_ipcw_scorer(model, tau=tau)
+search = GridSearchCV(
+    wrapped,
+    {"estimator__coxphsurvivalanalysis__alpha": [0.01, 0.1, 1.0]},
+    cv=inner_splits,
+)
+```
+
+The wrapper learns the censoring distribution from each fit fold. Prefix wrapped
+parameters with `estimator__`. Enable scikit-learn metadata routing only when
+passing extra metadata through a meta-estimator. For example, Coxnet's
+`set_predict_request(alpha=True)` matters only when routing the `alpha` prediction
+argument with `sklearn.set_config(enable_metadata_routing=True)`.
+
+Use an outer CV loop for an unbiased CV performance estimate after inner tuning.
+Do not select parameters and report performance from the same folds as if external.
+
+## Competing risks
+
+```python
+from sksurv.nonparametric import cumulative_incidence_competing_risks
+
+# status: integer array, 0=censored, 1..K=mutually exclusive causes
+time, cif = cumulative_incidence_competing_risks(status, observed_time)
+total_cif = cif[0]
+cause_1_cif = cif[1]
+```
+
+`cif` has shape `(K + 1, n_times)`; row 0 is total risk and rows 1..K are
+cause-specific cumulative incidence. Cause-specific Cox models treat other causes
+as censored to estimate cause-specific hazards, but one such model's
+`1 - survival` is not the cause-specific CIF. See `references/competing-risks.md`.
+
+## Bundled local CLIs
+
+All helpers use deterministic synthetic data when no input is given. They make no
+network calls, reject URLs and symlinks, bound files/rows/features, avoid unsafe
+pickle loading, and lazily import scientific packages.
+
+```bash
+python skills/scikit-survival/scripts/validate_survival_csv.py --help
+python skills/scikit-survival/scripts/train_survival_model.py --help
+python skills/scikit-survival/scripts/evaluate_survival_metrics.py --help
+python skills/scikit-survival/scripts/competing_risk_cif.py --help
+python skills/scikit-survival/scripts/model_report.py --help
+```
+
+Typical local flow:
+
+```bash
+python skills/scikit-survival/scripts/validate_survival_csv.py \
+  --input data.csv --event-column event --time-column time \
+  --feature-columns age,group,measurement --structured-output outcome.npy
+
+python skills/scikit-survival/scripts/train_survival_model.py \
+  --input data.csv --event-column event --time-column time \
+  --numeric-columns age,measurement --categorical-columns group \
+  --model coxph --tune --prediction-output predictions.npz \
+  --output training-summary.json
+
+python skills/scikit-survival/scripts/evaluate_survival_metrics.py \
+  --input predictions.npz --output metrics-summary.json
+
+python skills/scikit-survival/scripts/model_report.py \
+  --training-summary training-summary.json \
+  --metrics-summary metrics-summary.json --output model-report.md
+```
+
+Use only de-identified, authorized local data. The bundled tests contain synthetic
+records only and no patient data or PHI.
+
+## Security triage
+
+`SECURITY.md` previously claimed this skill bundled package-shadowing files named
+`sklearn.py` and `sksurv.py`. The 2026-07-23 inventory confirmed those files did
+not exist; the claim was a phantom analyzer finding. This refresh adds only
+descriptively named helpers and no shadow modules, environment reads, or network
+calls.
+
+Never name a project script after an imported package (including `sklearn.py`,
+`sksurv.py`, `numpy.py`, or `pandas.py`), because Python may import the local file
+instead of the installed library. Inspect the working directory before executing
+examples copied from untrusted sources.
+
+## Reference files
+
+- `references/data-handling.md` — structured arrays, datasets, schema validation,
+  pandas/Polars preprocessing, and leakage-safe splitting.
+- `references/cox-models.md` — Cox PH, Coxnet, IPCRidge, assumptions, and tuning.
+- `references/ensemble-models.md` — forests, trees, boosting, predictions, and
+  permutation importance.
+- `references/svm-models.md` — SVM objectives, prediction direction, scaling,
+  kernels, and limitations.
+- `references/evaluation-metrics.md` — metric inputs, censoring assumptions,
+  time grids, calibration, nested CV, and primary literature.
+- `references/competing-risks.md` — integer event coding, CIF API, built-in
+  datasets, cause-specific hazards, and unsupported Fine-Gray regression.
+
+## Dated sources
+
+Official API and compatibility sources, checked 2026-07-23:
+
+- [PyPI 0.28.0](https://pypi.org/project/scikit-survival/) — released 2026-07-05.
+- [GitHub v0.28.0 release](https://github.com/sebp/scikit-survival/releases/tag/v0.28.0)
+  — published 2026-07-05.
+- [0.28 release notes](https://scikit-survival.readthedocs.io/en/stable/release_notes/v0.28.html).
+- [Installation guide](https://scikit-survival.readthedocs.io/en/stable/install.html).
+- [Stable user guide](https://scikit-survival.readthedocs.io/en/stable/user_guide/index.html).
+- [Stable API reference](https://scikit-survival.readthedocs.io/en/stable/api/index.html).

package/skills/scikit-survival/references/competing-risks.md

@@ -0,0 +1,302 @@
+# Competing risks and cumulative incidence
+
+Verified for scikit-survival 0.28.0 on 2026-07-23.
+
+## Estimand
+
+Competing risks are mutually exclusive causes \(J \in \{1,\ldots,K\}\), where the
+first observed cause prevents observing the others as first events.
+
+The cause-\(k\) cumulative incidence function (CIF) is:
+
+\[
+F_k(t) = P(T \le t, J=k).
+\]
+
+It is an absolute cause-specific event probability accounting for all competing
+causes. It is not:
+
+- a cause-specific hazard;
+- `1 - Kaplan-Meier` after censoring other causes;
+- a conditional probability among only those still event-free;
+- a causal effect or clinical-utility measure.
+
+The total risk is \(\sum_k F_k(t)\). Its complement is estimated all-cause
+event-free survival. Censoring is an observation mechanism, not an additional
+event-free state.
+
+## Event coding
+
+The nonparametric CIF API takes two separate arrays:
+
+```python
+# event: 0=censored; 1..K=mutually exclusive causes
+event = frame["status"].to_numpy(dtype=int)
+time = frame["time"].to_numpy(dtype=float)
+```
+
+Requirements:
+
+- `event` is integer and non-negative;
+- 0 always denotes right-censoring;
+- positive codes 1..K are contiguous;
+- the data contains observations for every code 1..K;
+- `time` is finite and positive;
+- event/time lengths match.
+
+Do not pass a boolean `Surv` outcome to
+`cumulative_incidence_competing_risks()`. `Surv` intentionally collapses event
+status to event versus censoring and loses cause identity.
+
+## Nonparametric CIF API
+
+```python
+from sksurv.nonparametric import cumulative_incidence_competing_risks
+
+time_points, cumulative_incidence = (
+    cumulative_incidence_competing_risks(event, time)
+)
+```
+
+Current signature:
+
+```text
+cumulative_incidence_competing_risks(
+    event,
+    time_exit,
+    time_min=None,
+    conf_level=0.95,
+    conf_type=None,
+    var_type="Aalen",
+)
+```
+
+Returns:
+
+- `time_points`: shape `(n_times,)`;
+- `cumulative_incidence`: shape `(K + 1, n_times)`;
+- row 0: total risk of any cause;
+- row `k`: CIF for cause `k`.
+
+```python
+total_risk = cumulative_incidence[0]
+cause_1 = cumulative_incidence[1]
+cause_2 = cumulative_incidence[2]
+
+assert np.allclose(
+    total_risk,
+    cumulative_incidence[1:].sum(axis=0),
+)
+```
+
+`time_min` estimates conditionally on surviving at least to that time. This changes
+the target population and must not be selected after viewing outcomes.
+
+### Confidence intervals
+
+```python
+time_points, cumulative_incidence, confidence_interval = (
+    cumulative_incidence_competing_risks(
+        event,
+        time,
+        conf_type="log-log",
+        conf_level=0.95,
+        var_type="Aalen",
+    )
+)
+```
+
+`confidence_interval` has shape `(K + 1, 2, n_times)`, where axis 1 is lower/upper.
+Current variance choices are:
+
+- `"Aalen"`
+- `"Dinse"`
+- `"Dinse_Approx"`
+
+Pointwise confidence intervals are not simultaneous confidence bands. Sparse
+causes and late follow-up can make estimates unstable even when the function
+returns a result.
+
+## Built-in competing-risk datasets
+
+```python
+from sksurv.datasets import load_bmt, load_cgvhd
+
+X_bmt, y_bmt = load_bmt()       # status codes 0, 1, 2
+X_cgvhd, y_cgvhd = load_cgvhd() # status codes 0, 1, 2, 3
+```
+
+The first structured field is integer cause status, not boolean. These are real
+study datasets distributed for examples. The bundled tests do not use them; they
+use synthetic non-clinical outcomes only.
+
+## Why `1 - Kaplan-Meier` is wrong for one cause
+
+If cause 2 prevents cause 1, censoring cause 2 in a Kaplan-Meier curve treats those
+subjects as if they could still experience cause 1 later under non-informative
+censoring. That counterfactual risk set does not estimate the observed-world
+probability \(F_1(t)\) and typically overstates cause-1 probability.
+
+Use CIF for cause-specific absolute probability:
+
+```python
+time_points, cif = cumulative_incidence_competing_risks(event, time)
+probability_cause_1_by_t = cif[1]
+```
+
+Kaplan-Meier remains appropriate for all-cause event-free survival after collapsing
+all causes to event, if that is the estimand and censoring assumptions hold.
+
+## Comparing groups
+
+Estimate group-specific curves without fitting preprocessing on the full dataset:
+
+```python
+curves = {}
+for label in prespecified_groups:
+    mask = group == label
+    curves[label] = cumulative_incidence_competing_risks(
+        event[mask],
+        time[mask],
+        conf_type="log-log",
+    )
+```
+
+Plotting pointwise intervals does not test equality. scikit-survival 0.28 does not
+provide Gray's test in this API. Do not substitute an ordinary log-rank test:
+survival and CIF group hypotheses differ.
+
+Group labels and comparison times should be prespecified. Report at-risk/event
+support; late visual separation with few rows can be misleading.
+
+## Cause-specific Cox hazards
+
+For cause \(k\), a cause-specific hazard model encodes that cause as an event and
+other causes as censored at their occurrence time:
+
+```python
+from sksurv.linear_model import CoxPHSurvivalAnalysis
+from sksurv.util import Surv
+
+y_cause_1 = Surv.from_arrays(
+    event=(event == 1),
+    time=time,
+)
+cause_1_hazard_model = CoxPHSurvivalAnalysis(alpha=0.1)
+cause_1_hazard_model.fit(X_train, y_cause_1_train)
+```
+
+This estimates association with the instantaneous cause-specific hazard under a
+PH model. Other causes are censored for this hazard likelihood, which is different
+from pretending they are independent censoring when estimating absolute CIF.
+
+To derive cause-specific CIF predictions from cause-specific hazards, all modeled
+causes must be combined:
+
+\[
+F_k(t \mid x) =
+\int_0^t S(u^- \mid x)\,dH_k(u \mid x),
+\quad
+S(t \mid x)=\exp\left[-\sum_j H_j(t \mid x)\right].
+\]
+
+Therefore, `1 - cause_1_model.predict_survival_function(...)` is not the cause-1
+CIF. A set of separately fitted cause-specific models requires careful joint
+integration, common time grids, and external validation.
+
+## Fine-Gray regression
+
+scikit-survival 0.28 does not implement Fine-Gray subdistribution-hazard
+regression. Do not invent an import or describe `cumulative_incidence_competing_risks`
+as Fine-Gray; it is a nonparametric CIF estimator.
+
+If using another implementation:
+
+- verify it is actively maintained and supports the required censoring/truncation;
+- use its official API documentation;
+- distinguish subdistribution from cause-specific hazard coefficients;
+- keep preprocessing and tuning leakage-safe;
+- validate cause-specific absolute probabilities, not only coefficients.
+
+Neither hazard parameterization is universally "better." The estimand determines
+the method.
+
+## Prediction evaluation
+
+Standard `concordance_index_ipcw`, `cumulative_dynamic_auc`, and Brier APIs in
+scikit-survival are documented for right-censored single-event outcomes. A
+competing-risk prediction question needs:
+
+- a named cause;
+- a case/control definition at each horizon;
+- handling of other causes consistent with that definition;
+- cause-specific probability predictions for calibration/Brier evaluation;
+- censoring weights fitted on training data;
+- evaluation times supported by training follow-up;
+- nested tuning or an untouched holdout.
+
+Do not label an all-event C-index as cause-specific discrimination, and do not use
+an all-event survival probability as a cause-specific CIF.
+
+## Bundled helper
+
+The helper defaults to deterministic synthetic data:
+
+```bash
+python skills/scikit-survival/scripts/competing_risk_cif.py
+```
+
+For local CSV:
+
+```bash
+python skills/scikit-survival/scripts/competing_risk_cif.py \
+  --input competing.csv \
+  --event-column status \
+  --time-column time \
+  --horizons 2,5,10 \
+  --confidence \
+  --curve-output cif-curves.npz \
+  --output cif-summary.json
+```
+
+It:
+
+- rejects URLs, symlinks, missing/non-contiguous causes, and invalid times;
+- bounds file size and row count;
+- verifies that cause-specific rows sum to total CIF;
+- writes numeric arrays without pickle;
+- reports point estimates at requested horizons;
+- makes no network calls.
+
+Use only authorized, de-identified local data. Do not include row-level data or PHI
+in reports.
+
+## Reporting checklist
+
+- cause definitions and code mapping;
+- censoring definition and follow-up window;
+- CIF versus cause-specific or subdistribution hazard estimand;
+- number of rows/events for every cause;
+- horizon-specific CIF with uncertainty and support;
+- whether intervals are pointwise;
+- handling of `time_min`, if any;
+- competing-risk-specific prediction evaluation;
+- no causal or clinical-utility claim from association/probability alone.
+
+## Sources
+
+Official scikit-survival sources checked 2026-07-23:
+
+- [Competing-risks user guide](https://scikit-survival.readthedocs.io/en/stable/user_guide/competing-risks.html)
+- [CIF API](https://scikit-survival.readthedocs.io/en/stable/api/generated/sksurv.nonparametric.cumulative_incidence_competing_risks.html)
+- [Dataset API](https://scikit-survival.readthedocs.io/en/stable/api/datasets.html)
+- [0.24 release notes introducing CIF](https://scikit-survival.readthedocs.io/en/stable/release_notes/v0.24.html)
+
+Primary methods:
+
+- Aalen O. "Nonparametric estimation of partial transition probabilities in
+  multiple decrement models." *Annals of Statistics* 6 (1978), 534-545.
+  [Project Euclid record](https://projecteuclid.org/journals/annals-of-statistics/volume-6/issue-3/Nonparametric-Estimation-of-Partial-Transition-Probabilities-in-Multiple-Decrement-Models/10.1214/aos/1176344198.full)
+- Gray RJ. "A class of K-sample tests for comparing the cumulative incidence of
+  a competing risk." *Annals of Statistics* 16 (1988), 1141-1154.
+  [doi:10.1214/aos/1176350951](https://doi.org/10.1214/aos/1176350951)