firthmodels 0.8.0__tar.gz → 0.8.2__tar.gz

{firthmodels-0.8.0 → firthmodels-0.8.2}/.pre-commit-config.yaml

@@ -7,7 +7,7 @@ repos:
     -   id: trailing-whitespace
 - repo: https://github.com/astral-sh/ruff-pre-commit
   # Ruff version.
-  rev: v0.14.7
+  rev: v0.14.8
   hooks:
     # Run the linter.
     - id: ruff-check

{firthmodels-0.8.0 → firthmodels-0.8.2}/CITATION.cff

@@ -2,7 +2,7 @@ cff-version: 1.2.0
 title: firthmodels
 message: "If you use this software, please cite it as below."
 type: software
-version: 0.8.0
+version: 0.8.2
 doi: 10.5281/zenodo.17863280
 authors:
   - family-names: Luo

{firthmodels-0.8.0 → firthmodels-0.8.2}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.4
+Metadata-Version: 2.5
 Name: firthmodels
-Version: 0.8.0
+Version: 0.8.2
 Summary: Firth-penalized models in Python
 Project-URL: Homepage, https://github.com/jzluo/firthmodels
 Project-URL: Repository, https://github.com/jzluo/firthmodels

firthmodels-0.8.2/benchmarks/README.md

@@ -0,0 +1,191 @@
+# Benchmarks
+
+Benchmarking of [firthmodels](https://github.com/jzluo/firthmodels) against implementations
+of Firth-penalized logistic regression (R [logistf](https://cran.r-project.org/package=logistf),
+[brglm2](https://cran.r-project.org/package=brglm2)) and Cox regression
+(R [coxphf](https://cran.r-project.org/package=coxphf)).
+
+## Summary
+
+For the full workflow, firthmodels' Numba backend is **10.5x
+faster than logistf** (k=50) and **158x faster than
+coxphf** (k=30). Without Numba, the pure NumPy backend is 7.3x
+faster than logistf and 23.4x faster than coxphf.
+
+| Workload | Baseline | Speedup at k=5 | Speedup at largest k |
+|---|---|--:|--:|
+| Logistic: fit + Wald | next-fastest package | 3.7x | 5.2x |
+| Logistic: fit + LRT + profile CI | logistf | 2.5x | 10.5x |
+| Cox: fit + Wald | coxphf | 11.2x | 130x |
+| Cox: fit + LRT + profile CI | coxphf | 8.1x | 158x |
+
+The largest observed deviation across coefficients, profile CI bounds, and p-values is 8.0e-07 (see [Correctness](#correctness)).
+
+Python is timed with time.perf_counter() around each call, after JIT-warming the Numba
+backend. R packages are timed in-process with `microbenchmark` inside a single
+R session, so R startup and data transfer are excluded, while formula parsing and
+model-frame construction are included because logistf and coxphf only
+offer formula interfaces. brglm2 is run with `check_aliasing=FALSE` to avoid
+extra overhead from its default aliasing check. The reported value is the fastest of 10 runs.
+
+---
+
+## Firth logistic regression
+
+Compared against R [logistf](https://cran.r-project.org/package=logistf) and
+[brglm2](https://cran.r-project.org/package=brglm2) on simulated data with
+n = 1,000 observations, a 20% target event rate, and
+k = 5 to 50 features.
+
+![Logistic benchmark scaling, log time axis](logistic_results.png)
+
+The time axis is log-scale, so a constant vertical gap means a constant speedup ratio.
+All values are the minimum observed wall-clock time across repeated runs, in milliseconds.
+
+### Fit + Wald inference
+
+| k | firthmodels<br>(numba) | firthmodels<br>(numpy) | logistf | brglm2<br>(AS-mean) | brglm2<br>(MPL-Jeffreys) | numba speedup<br>vs next fastest | numpy speedup<br>vs next fastest |
+|--:|--:|--:|--:|--:|--:|--:|--:|
+| 5 | **0.53** | 0.92 | <ins>2.0</ins> | 4.1 | 4.1 | 3.7x | 2.1x |
+| 10 | **0.74** | 1.2 | <ins>3.5</ins> | 4.8 | 4.8 | 4.7x | 2.8x |
+| 15 | **0.77** | 1.3 | <ins>5.4</ins> | 5.6 | 5.5 | 7.0x | 4.2x |
+| 20 | **1.1** | 1.6 | 8.0 | 6.4 | <ins>6.4</ins> | 6.1x | 4.0x |
+| 25 | **1.5** | 2.3 | 11.7 | 8.1 | <ins>8.0</ins> | 5.4x | 3.6x |
+| 30 | **1.7** | 2.5 | 15.1 | <ins>8.8</ins> | 8.9 | 5.3x | 3.5x |
+| 35 | **1.8** | 2.8 | 20.3 | 10.5 | <ins>10.5</ins> | 5.9x | 3.7x |
+| 40 | **2.1** | 3.2 | 25.3 | 11.9 | <ins>11.8</ins> | 5.5x | 3.7x |
+| 45 | **2.8** | 3.8 | 33.4 | 13.3 | <ins>13.2</ins> | 4.8x | 3.5x |
+| 50 | **3.3** | 4.4 | 41.5 | 17.3 | <ins>17.1</ins> | 5.2x | 3.9x |
+
+The fastest time at each k is **bolded**. The next-fastest package (not including the firthmodels
+NumPy backend) is <ins>underlined</ins> and is the baseline for the speedup columns. The two brglm2
+fitting methods have nearly identical timings (within 1.1% at every k).
+
+### Full workflow: fit + LRT + profile likelihood CI
+
+brglm2 is not included here because it does not provide penalized LRT
+p-values or profile likelihood CIs.
+
+| k | firthmodels<br>(numba) | firthmodels<br>(numpy) | logistf | numba speedup<br>vs logistf | numpy speedup<br>vs logistf |
+|--:|--:|--:|--:|--:|--:|
+| 5 | **4.8** | 11.3 | <ins>12.0</ins> | 2.5x | 1.1x |
+| 10 | **11.9** | 25.3 | <ins>42.4</ins> | 3.5x | 1.7x |
+| 15 | **19.1** | 40.3 | <ins>102.8</ins> | 5.4x | 2.6x |
+| 20 | **36.4** | 64.6 | <ins>209.1</ins> | 5.7x | 3.2x |
+| 25 | **68.5** | 120.3 | <ins>418.9</ins> | 6.1x | 3.5x |
+| 30 | **95.4** | 165.7 | <ins>629.5</ins> | 6.6x | 3.8x |
+| 35 | **115.4** | 208.8 | <ins>1,054</ins> | 9.1x | 5.0x |
+| 40 | **162.8** | 269.2 | <ins>1,531</ins> | 9.4x | 5.7x |
+| 45 | **222.1** | 334.8 | <ins>2,226</ins> | 10.0x | 6.6x |
+| 50 | **299.2** | 429.5 | <ins>3,128</ins> | 10.5x | 7.3x |
+
+---
+
+## Firth Cox proportional hazards
+
+Compared against R [coxphf](https://cran.r-project.org/package=coxphf) on
+simulated survival data with n = 500 observations, a 20% event rate,
+and k = 5 to 30 features.
+
+![Cox benchmark scaling, log time axis](cox_results.png)
+
+The time axis is log-scale, so a constant vertical gap means a constant speedup ratio.
+All values are the minimum observed wall-clock time across repeated runs, in milliseconds.
+
+### Fit + Wald inference
+
+| k | firthmodels<br>(numba) | firthmodels<br>(numpy) | coxphf | numba speedup<br>vs coxphf | numpy speedup<br>vs coxphf |
+|--:|--:|--:|--:|--:|--:|
+| 5 | **0.30** | 1.5 | <ins>3.4</ins> | 11.2x | 2.2x |
+| 10 | **0.41** | 2.3 | <ins>9.0</ins> | 22.0x | 4.0x |
+| 15 | **0.57** | 3.3 | <ins>24.9</ins> | 43.4x | 7.5x |
+| 20 | **0.86** | 5.7 | <ins>56.1</ins> | 65.4x | 9.8x |
+| 25 | **1.2** | 7.9 | <ins>122.2</ins> | 98.7x | 15.6x |
+| 30 | **1.6** | 10.5 | <ins>211.6</ins> | 130x | 20.2x |
+
+The fastest time at each k is **bolded**. The next-fastest package (not including the firthmodels
+NumPy backend) is <ins>underlined</ins> and is the baseline for the speedup columns.
+
+### Full workflow: fit + LRT + profile likelihood CI
+
+| k | firthmodels<br>(numba) | firthmodels<br>(numpy) | coxphf | numba speedup<br>vs coxphf | numpy speedup<br>vs coxphf |
+|--:|--:|--:|--:|--:|--:|
+| 5 | **2.2** | 12.8 | <ins>17.9</ins> | 8.1x | 1.4x |
+| 10 | **7.1** | 45.6 | <ins>163.0</ins> | 22.9x | 3.6x |
+| 15 | **17.4** | 114.1 | <ins>928.5</ins> | 53.4x | 8.1x |
+| 20 | **30.8** | 227.2 | <ins>2,547</ins> | 82.6x | 11.2x |
+| 25 | **59.4** | 404.1 | <ins>7,418</ins> | 125x | 18.4x |
+| 30 | **98.1** | 660.6 | <ins>15,467</ins> | 158x | 23.4x |
+
+---
+
+## Correctness
+
+The benchmark scripts abort if firthmodels disagrees with the R reference by more than
+1e-06 on coefficients, profile CI bounds, or p-values (and also cross-check
+the numba backend against the numpy backend).
+
+Maximum over all coefficients and all k during the benchmark run (recorded in the results file's run metadata):
+
+| Comparison | Quantity | Max abs. deviation |
+|---|---|--:|
+| Logistic fit vs logistf | Coefficients | 2.2e-16 |
+| Logistic fit vs brglm2 AS_mean | Coefficients | 7.3e-07 |
+| Logistic fit vs brglm2 MPL_Jeffreys | Coefficients | 7.3e-07 |
+| Logistic full vs logistf | Coefficients | 2.2e-16 |
+| Logistic full vs logistf | Profile CI bounds | 5.4e-13 |
+| Logistic full vs logistf | LRT p-values | 4.9e-10 |
+| Logistic numba vs numpy | Coefficients | 2.2e-16 |
+| Cox fit vs coxphf | Coefficients | 8.0e-07 |
+| Cox full vs coxphf | Coefficients | 8.0e-07 |
+| Cox full vs coxphf | Profile CI bounds | 4.3e-08 |
+| Cox full vs coxphf | LRT p-values | 2.6e-08 |
+| Cox numba vs numpy | Coefficients | 2.8e-16 |
+
+Note: the benchmark run(s) recorded firthmodels 0.8.1, while the currently installed version is 0.8.2. Timings describe the recorded version.
+
+---
+
+## Environment
+
+Collected at report-generation time on the benchmark machine.
+
+| Component | Version |
+|---|---|
+| OS | CachyOS |
+| CPU | AMD Ryzen 5 5600X 6-Core Processor |
+| Python | 3.12.13 |
+| firthmodels | 0.8.2 |
+| NumPy / SciPy / Numba | 2.3.5 / 1.16.3 / 0.64.0 |
+| NumPy BLAS | /usr/lib/libopenblas.so.0.3 (openblas 0.3.34, runtime) |
+| R | 4.6.1 |
+| logistf / brglm2 / coxphf | 1.26.1 / 1.1.0 / 1.13.4 |
+| R BLAS | /usr/lib/libopenblas.so.0.3 |
+
+NumPy and R link to the same BLAS library (`/usr/lib/libopenblas.so.0.3`). BLAS threading is left at library defaults for both stacks.
+R packages are compiled from source with `-march=x86-64-v3 -mtune=haswell -O3 -flto=auto`,
+so the R timings are not slowed by a conservative build.
+
+<details>
+<summary>Full R package compile flags (current configuration)</summary>
+
+```
+CFLAGS:  -march=x86-64-v3 -mtune=haswell -O3 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -mpclmul -g1 -ffile-prefix-map=/startdir/src=/usr/src/debug/r -flto=auto -ffat-lto-objects
+FCFLAGS: -O3 -march=x86-64-v3 -mtune=haswell
+```
+
+</details>
+
+---
+
+## Reproducing these results
+
+Requires R with the logistf, brglm2, coxphf, survival, microbenchmark, and
+jsonlite packages installed.
+
+```bash
+# Run both benchmarks (writes results JSON files holding the timings, R
+# reference values, and run metadata), then regenerate the plots and this
+# README
+benchmarks/run_benchmarks.sh
+```

{firthmodels-0.8.0 → firthmodels-0.8.2}/benchmarks/benchmark_cox.py

@@ -14,7 +14,7 @@ import platform as plat
 import subprocess
 import sys
 import tempfile
-import time
+from datetime import datetime, timezone
 from pathlib import Path
 from typing import Literal
 
@@ -29,7 +29,7 @@ from firthmodels import FirthCoxPH
 N_SAMPLES = 500
 EVENT_RATE = 0.20
 K_VALUES = list(range(5, 35, 5))
-N_RUNS = 15
+N_RUNS = 10
 BASE_SEED = 0
 
 # Solver parameters
@@ -43,8 +43,8 @@ COEF_TOL = 1e-6
 CI_TOL = 1e-6
 PVAL_TOL = 1e-6
 
-# Reduce coxphf benchmark runs for large k
-COXPHF_REDUCE_AFTER = 5
+# Reduce coxphf benchmark runs for large k (0 disables reduction)
+COXPHF_REDUCE_AFTER = 0
 
 
 # -----------------------------------------------------------------------------
@@ -279,26 +279,41 @@ def check_agreement(
     r_ci: np.ndarray | None = None,
     py_pval: np.ndarray | None = None,
     r_pval: np.ndarray | None = None,
+    deviations: dict[str, dict[str, float]] | None = None,
 ) -> None:
-    """Check that Python and R results agree within tolerance."""
-    # Check coefficients
+    """Check that Python and R results agree within tolerance.
+
+    If a deviations dict is provided, the observed max absolute differences
+    are merged into it under label, keyed by quantity, taking the max over
+    successive calls (i.e. over k). Recorded before any tolerance failure is
+    raised.
+    """
+    observed: dict[str, float] = {}
+
     coef_diff = np.max(np.abs(py_coef - r_coef))
-    if coef_diff > COEF_TOL:
-        raise AssertionError(f"k={k} ({label}): coef diff {coef_diff:.2e} > {COEF_TOL}")
+    observed["coef"] = float(coef_diff)
 
-    # Check CIs if provided
     if py_ci is not None and r_ci is not None:
-        ci_diff = np.max(np.abs(py_ci - r_ci))
-        if ci_diff > CI_TOL:
-            raise AssertionError(f"k={k} ({label}): CI diff {ci_diff:.2e} > {CI_TOL}")
+        observed["ci"] = float(np.max(np.abs(py_ci - r_ci)))
 
-    # Check p-values if provided
     if py_pval is not None and r_pval is not None:
-        pval_diff = np.max(np.abs(py_pval - r_pval))
-        if pval_diff > PVAL_TOL:
-            raise AssertionError(
-                f"k={k} ({label}): p-value diff {pval_diff:.2e} > {PVAL_TOL}"
-            )
+        observed["pval"] = float(np.max(np.abs(py_pval - r_pval)))
+
+    if deviations is not None:
+        bucket = deviations.setdefault(label, {})
+        for quantity, value in observed.items():
+            bucket[quantity] = max(bucket.get(quantity, 0.0), value)
+
+    if coef_diff > COEF_TOL:
+        raise AssertionError(f"k={k} ({label}): coef diff {coef_diff:.2e} > {COEF_TOL}")
+    if observed.get("ci", 0.0) > CI_TOL:
+        raise AssertionError(
+            f"k={k} ({label}): CI diff {observed['ci']:.2e} > {CI_TOL}"
+        )
+    if observed.get("pval", 0.0) > PVAL_TOL:
+        raise AssertionError(
+            f"k={k} ({label}): p-value diff {observed['pval']:.2e} > {PVAL_TOL}"
+        )
 
 
 # -----------------------------------------------------------------------------
@@ -426,17 +441,17 @@ def compute_min(times: np.ndarray) -> float:
 # Saved results loading
 # -----------------------------------------------------------------------------
 def load_saved_results(
-    saved_csv: str,
+    saved_path: str,
     k_values: list[int],
     load_firthmodels: bool = False,
     load_coxphf: bool = True,
 ) -> tuple[pd.DataFrame, dict[int, dict] | None]:
-    """Load results from a saved CSV file.
+    """Load results from a saved results JSON file.
 
     Parameters
     ----------
-    saved_csv : str
-        Path to CSV with previous results.
+    saved_path : str
+        Path to results JSON with previous results.
     k_values : list of int
         Which k values to load.
     load_firthmodels : bool
@@ -451,7 +466,8 @@ def load_saved_results(
     coxphf_results : dict or None
         coxphf results if load_coxphf=True, else None.
     """
-    saved_df = pd.read_csv(saved_csv)
+    with open(saved_path) as f:
+        saved_df = pd.DataFrame(json.load(f)["results"])
 
     # Determine required columns based on what we're loading
     required_cols = {"k", "n"}
@@ -473,20 +489,21 @@ def load_saved_results(
 
     missing_cols = required_cols - set(saved_df.columns)
     if missing_cols:
-        raise ValueError(f"Saved CSV missing columns: {missing_cols}")
+        raise ValueError(f"Saved results missing columns: {missing_cols}")
 
     saved_n = int(saved_df["n"].iloc[0])
     if saved_n != N_SAMPLES:
         raise ValueError(
-            f"Saved CSV was created with n={saved_n}, but current N_SAMPLES={N_SAMPLES}."
+            f"Saved results were created with n={saved_n}, "
+            f"but current N_SAMPLES={N_SAMPLES}."
         )
 
     saved_k = set(saved_df["k"].tolist())
     missing = set(k_values) - saved_k
     if missing:
-        raise ValueError(f"k values {missing} not in saved CSV")
+        raise ValueError(f"k values {missing} not in saved results")
 
-    # No need to parse JSON - we copy strings directly from saved_df rows
+    # Reference values are copied verbatim from saved_df rows, so no parsing
     return saved_df, None
 
 
@@ -501,8 +518,13 @@ def run_benchmarks(
     saved: str | None = None,
     run_firthmodels: bool = True,
     run_coxphf: bool = True,
-) -> tuple[pd.DataFrame, dict[str, str]]:
-    """Run all benchmarks and return results as DataFrame."""
+) -> tuple[pd.DataFrame, dict[str, str], dict]:
+    """Run all benchmarks and return results as DataFrame.
+
+    Also returns version_info and run_meta (run counts, verification status
+    and observed deviations, versions; saved into the results JSON by
+    main()).
+    """
     version_info = get_python_version_info()
 
     # Load saved results for libraries we're not running
@@ -580,7 +602,7 @@ def run_benchmarks(
         if coxphf_results is None:
             coxphf_results = {}
         if run_coxphf:
-            if coxphf_reduce_after is not None:
+            if coxphf_reduce_after:
                 k_low = [k for k in k_values if k <= coxphf_reduce_after]
                 k_high = [k for k in k_values if k > coxphf_reduce_after]
 
@@ -612,6 +634,7 @@ def run_benchmarks(
             "Verifying results and computing statistics...", file=sys.stderr, flush=True
         )
         results = []
+        deviations: dict[str, dict[str, float]] = {}
 
         for k in k_values:
             X, surv_time, event = make_benchmark_data(
@@ -622,21 +645,32 @@ def run_benchmarks(
             if run_firthmodels and run_coxphf and not skip_verification:
                 py = py_results["numba"][k]
                 r = coxphf_results[k]
-                check_agreement(k, py["fit_coef"], r["fit_coef"], "fit")
+                check_agreement(
+                    k,
+                    py["fit_coef"],
+                    r["fit_coef"],
+                    "fit vs coxphf",
+                    deviations=deviations,
+                )
                 check_agreement(
                     k,
                     py["full_coef"],
                     r["full_coef"],
-                    "full",
+                    "full vs coxphf",
                     py["full_ci"],
                     r["full_ci"],
                     py["full_pval"],
                     r["full_pval"],
+                    deviations=deviations,
                 )
                 # Also check numba vs numpy
                 py_numpy = py_results["numpy"][k]
                 check_agreement(
-                    k, py["fit_coef"], py_numpy["fit_coef"], "numba vs numpy"
+                    k,
+                    py["fit_coef"],
+                    py_numpy["fit_coef"],
+                    "numba vs numpy",
+                    deviations=deviations,
                 )
 
             # Collect timing results
@@ -673,10 +707,10 @@ def run_benchmarks(
                 row["coxphf_fit_ms"] = compute_min(r["fit_times"]) * 1000
                 row["coxphf_full_ms"] = compute_min(r["full_times"]) * 1000
                 # Store for reproducibility
-                row["coxphf_fit_coef"] = json.dumps(r["fit_coef"].tolist())
-                row["coxphf_full_coef"] = json.dumps(r["full_coef"].tolist())
-                row["coxphf_full_ci"] = json.dumps(r["full_ci"].tolist())
-                row["coxphf_full_pval"] = json.dumps(r["full_pval"].tolist())
+                row["coxphf_fit_coef"] = np.asarray(r["fit_coef"]).tolist()
+                row["coxphf_full_coef"] = np.asarray(r["full_coef"]).tolist()
+                row["coxphf_full_ci"] = np.asarray(r["full_ci"]).tolist()
+                row["coxphf_full_pval"] = np.asarray(r["full_pval"]).tolist()
             else:
                 assert saved_row is not None
                 row["coxphf_fit_ms"] = saved_row["coxphf_fit_ms"]
@@ -688,7 +722,39 @@ def run_benchmarks(
 
             results.append(row)
 
-    return pd.DataFrame(results), version_info
+    reduced = n_runs if n_runs <= 3 else max(3, n_runs // 3)
+    run_meta = {
+        "schema_version": 1,
+        "model": "cox",
+        "created_utc": datetime.now(timezone.utc).isoformat(timespec="seconds"),
+        "n_runs": n_runs,
+        "r_reduced_runs": (
+            {
+                "libraries": ["coxphf"],
+                "for_k_above": coxphf_reduce_after,
+                "n_runs": reduced,
+            }
+            if coxphf_reduce_after
+            else None
+        ),
+        "k_values": list(k_values),
+        "libraries_run": [
+            name
+            for name, flag in [
+                ("firthmodels", run_firthmodels),
+                ("coxphf", run_coxphf),
+            ]
+            if flag
+        ],
+        "loaded_from_saved": saved,
+        "verification": {
+            "performed": bool(deviations),
+            "tolerances": {"coef": COEF_TOL, "ci": CI_TOL, "pval": PVAL_TOL},
+            "max_abs_deviations": deviations,
+        },
+        "versions": version_info,
+    }
+    return pd.DataFrame(results), version_info, run_meta
 
 
 def print_table(df: pd.DataFrame) -> None:
@@ -743,14 +809,14 @@ def main():
         "--out",
         type=str,
         default=None,
-        help="Save results to CSV file",
+        help="Save results (timings, R references, run metadata) to JSON file",
     )
     parser.add_argument(
         "--saved",
         type=str,
         default=None,
-        metavar="CSV",
-        help="Load non-selected libraries from this CSV.",
+        metavar="JSON",
+        help="Load non-selected libraries from this results JSON.",
     )
     parser.add_argument(
         "--firthmodels",
@@ -807,7 +873,7 @@ def main():
         )
 
     # Run benchmarks
-    df, version_info = run_benchmarks(
+    df, version_info, run_meta = run_benchmarks(
         k_values=k_values,
         n_runs=args.n_runs,
         skip_verification=args.skip_verification,
@@ -821,7 +887,11 @@ def main():
     print_table(df)
 
     if args.out:
-        df.to_csv(args.out, index=False)
+        with open(args.out, "w") as f:
+            json.dump(
+                {"meta": run_meta, "results": df.to_dict(orient="records")}, f, indent=2
+            )
+            f.write("\n")
         print(f"\nResults saved to {args.out}", file=sys.stderr)