firthmodels 0.7.2__tar.gz → 0.8.1__tar.gz

{firthmodels-0.7.2 → firthmodels-0.8.1}/CITATION.cff

@@ -2,7 +2,7 @@ cff-version: 1.2.0
 title: firthmodels
 message: "If you use this software, please cite it as below."
 type: software
-version: 0.7.2
+version: 0.8.1
 doi: 10.5281/zenodo.17863280
 authors:
   - family-names: Luo

{firthmodels-0.7.2 → firthmodels-0.8.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: firthmodels
-Version: 0.7.2
+Version: 0.8.1
 Summary: Firth-penalized models in Python
 Project-URL: Homepage, https://github.com/jzluo/firthmodels
 Project-URL: Repository, https://github.com/jzluo/firthmodels
@@ -228,6 +228,10 @@ model.lrt_bse_         # Back-corrected standard errors (separate from Wald bse_
 Each feature requires a separate constrained model fit, so you can test selectively to
 avoid unnecessary computation. By default, LRT uses a warm start based on the full-model
 covariance to reduce Newton-Raphson iterations; pass `warm_start=False` to disable it.
+If the full-model fit does not converge, its likelihood is non-finite, or a constrained
+fit does not converge, a warning is emitted and the affected p-values and
+back-corrected standard errors remain `NaN`. A later explicit `lrt()` call retries
+failed constrained fits.
 
 ```python
 model.lrt(0)              # Single feature by index

{firthmodels-0.7.2 → firthmodels-0.8.1}/README.md

@@ -185,6 +185,10 @@ model.lrt_bse_         # Back-corrected standard errors (separate from Wald bse_
 Each feature requires a separate constrained model fit, so you can test selectively to
 avoid unnecessary computation. By default, LRT uses a warm start based on the full-model
 covariance to reduce Newton-Raphson iterations; pass `warm_start=False` to disable it.
+If the full-model fit does not converge, its likelihood is non-finite, or a constrained
+fit does not converge, a warning is emitted and the affected p-values and
+back-corrected standard errors remain `NaN`. A later explicit `lrt()` call retries
+failed constrained fits.
 
 ```python
 model.lrt(0)              # Single feature by index

firthmodels-0.8.1/benchmarks/README.md

@@ -0,0 +1,189 @@
+# Benchmarks
+
+Benchmarking of [firthmodels](https://github.com/jzluo/firthmodels) against implementations
+of Firth-penalized logistic regression (R [logistf](https://cran.r-project.org/package=logistf),
+[brglm2](https://cran.r-project.org/package=brglm2)) and Cox regression
+(R [coxphf](https://cran.r-project.org/package=coxphf)).
+
+## Summary
+
+For the full workflow, firthmodels' Numba backend is **10.8x
+faster than logistf** (k=50) and **159x faster than
+coxphf** (k=30). Without Numba, the pure NumPy backend is 7.5x
+faster than logistf and 23.7x faster than coxphf.
+
+| Workload | Baseline | Speedup at k=5 | Speedup at largest k |
+|---|---|--:|--:|
+| Logistic: fit + Wald | next-fastest package | 3.8x | 5.3x |
+| Logistic: fit + LRT + profile CI | logistf | 2.7x | 10.8x |
+| Cox: fit + Wald | coxphf | 11.4x | 132x |
+| Cox: fit + LRT + profile CI | coxphf | 8.4x | 159x |
+
+The largest observed deviation across coefficients, profile CI bounds, and p-values is 8.0e-07 (see [Correctness](#correctness)).
+
+Python is timed with time.perf_counter() around each call, after JIT-warming the Numba
+backend. R packages are timed in-process with `microbenchmark` inside a single
+R session, so R startup and data transfer are excluded, while formula parsing and
+model-frame construction are included because logistf and coxphf only
+offer formula interfaces. brglm2 is run with `check_aliasing=FALSE` to avoid
+extra overhead from its default aliasing check. The reported value is the fastest of 10 runs.
+
+---
+
+## Firth logistic regression
+
+Compared against R [logistf](https://cran.r-project.org/package=logistf) and
+[brglm2](https://cran.r-project.org/package=brglm2) on simulated data with
+n = 1,000 observations, a 20% target event rate, and
+k = 5 to 50 features.
+
+![Logistic benchmark scaling, log time axis](logistic_results.png)
+
+The time axis is log-scale, so a constant vertical gap means a constant speedup ratio.
+All values are the minimum observed wall-clock time across repeated runs, in milliseconds.
+
+### Fit + Wald inference
+
+| k | firthmodels<br>(numba) | firthmodels<br>(numpy) | logistf | brglm2<br>(AS-mean) | brglm2<br>(MPL-Jeffreys) | numba speedup<br>vs next fastest | numpy speedup<br>vs next fastest |
+|--:|--:|--:|--:|--:|--:|--:|--:|
+| 5 | **0.50** | 0.87 | <ins>1.9</ins> | 4.0 | 4.0 | 3.8x | 2.2x |
+| 10 | **0.72** | 1.1 | <ins>3.4</ins> | 4.6 | 4.6 | 4.7x | 3.0x |
+| 15 | **0.75** | 1.2 | <ins>5.3</ins> | 5.4 | 5.4 | 7.1x | 4.4x |
+| 20 | **1.0** | 1.5 | 7.8 | 6.3 | <ins>6.2</ins> | 6.1x | 4.1x |
+| 25 | **1.5** | 2.2 | 11.4 | 7.8 | <ins>7.8</ins> | 5.4x | 3.6x |
+| 30 | **1.6** | 2.3 | 14.7 | <ins>8.5</ins> | 8.7 | 5.2x | 3.7x |
+| 35 | **1.7** | 2.6 | 19.9 | <ins>10.2</ins> | 10.2 | 5.8x | 4.0x |
+| 40 | **2.2** | 3.1 | 24.9 | 11.6 | <ins>11.5</ins> | 5.2x | 3.8x |
+| 45 | **2.7** | 3.7 | 33.2 | 13.1 | <ins>13.0</ins> | 4.8x | 3.5x |
+| 50 | **3.2** | 4.3 | 40.8 | 17.0 | <ins>16.7</ins> | 5.3x | 3.9x |
+
+The fastest time at each k is **bolded**. The next-fastest package (not including the firthmodels
+NumPy backend) is <ins>underlined</ins> and is the baseline for the speedup columns. The two brglm2
+fitting methods have nearly identical timings (within 1.9% at every k).
+
+### Full workflow: fit + LRT + profile likelihood CI
+
+brglm2 is not included here because it does not provide penalized LRT
+p-values or profile likelihood CIs.
+
+| k | firthmodels<br>(numba) | firthmodels<br>(numpy) | logistf | numba speedup<br>vs logistf | numpy speedup<br>vs logistf |
+|--:|--:|--:|--:|--:|--:|
+| 5 | **4.5** | 10.6 | <ins>11.9</ins> | 2.7x | 1.1x |
+| 10 | **11.5** | 23.7 | <ins>41.9</ins> | 3.6x | 1.8x |
+| 15 | **18.2** | 38.1 | <ins>101.0</ins> | 5.6x | 2.7x |
+| 20 | **34.9** | 61.7 | <ins>206.2</ins> | 5.9x | 3.3x |
+| 25 | **66.4** | 117.1 | <ins>418.7</ins> | 6.3x | 3.6x |
+| 30 | **92.9** | 150.3 | <ins>624.0</ins> | 6.7x | 4.2x |
+| 35 | **111.8** | 190.2 | <ins>1,059</ins> | 9.5x | 5.6x |
+| 40 | **166.4** | 261.9 | <ins>1,533</ins> | 9.2x | 5.9x |
+| 45 | **213.9** | 326.7 | <ins>2,230</ins> | 10.4x | 6.8x |
+| 50 | **290.9** | 420.3 | <ins>3,135</ins> | 10.8x | 7.5x |
+
+---
+
+## Firth Cox proportional hazards
+
+Compared against R [coxphf](https://cran.r-project.org/package=coxphf) on
+simulated survival data with n = 500 observations, a 20% event rate,
+and k = 5 to 30 features.
+
+![Cox benchmark scaling, log time axis](cox_results.png)
+
+The time axis is log-scale, so a constant vertical gap means a constant speedup ratio.
+All values are the minimum observed wall-clock time across repeated runs, in milliseconds.
+
+### Fit + Wald inference
+
+| k | firthmodels<br>(numba) | firthmodels<br>(numpy) | coxphf | numba speedup<br>vs coxphf | numpy speedup<br>vs coxphf |
+|--:|--:|--:|--:|--:|--:|
+| 5 | **0.29** | 1.4 | <ins>3.3</ins> | 11.4x | 2.3x |
+| 10 | **0.40** | 2.2 | <ins>8.7</ins> | 21.9x | 4.0x |
+| 15 | **0.57** | 3.2 | <ins>24.1</ins> | 42.5x | 7.5x |
+| 20 | **0.85** | 5.6 | <ins>54.8</ins> | 64.1x | 9.8x |
+| 25 | **1.2** | 7.7 | <ins>119.6</ins> | 99.8x | 15.5x |
+| 30 | **1.6** | 10.3 | <ins>210.3</ins> | 132x | 20.5x |
+
+The fastest time at each k is **bolded**. The next-fastest package (not including the firthmodels
+NumPy backend) is <ins>underlined</ins> and is the baseline for the speedup columns.
+
+### Full workflow: fit + LRT + profile likelihood CI
+
+| k | firthmodels<br>(numba) | firthmodels<br>(numpy) | coxphf | numba speedup<br>vs coxphf | numpy speedup<br>vs coxphf |
+|--:|--:|--:|--:|--:|--:|
+| 5 | **2.1** | 11.9 | <ins>17.4</ins> | 8.4x | 1.5x |
+| 10 | **6.8** | 43.6 | <ins>160.0</ins> | 23.4x | 3.7x |
+| 15 | **17.2** | 111.3 | <ins>912.8</ins> | 53.0x | 8.2x |
+| 20 | **31.4** | 221.8 | <ins>2,516</ins> | 80.2x | 11.3x |
+| 25 | **58.5** | 398.4 | <ins>7,281</ins> | 124x | 18.3x |
+| 30 | **97.1** | 651.4 | <ins>15,425</ins> | 159x | 23.7x |
+
+---
+
+## Correctness
+
+The benchmark scripts abort if firthmodels disagrees with the R reference by more than
+1e-06 on coefficients, profile CI bounds, or p-values (and also cross-check
+the numba backend against the numpy backend).
+
+Maximum over all coefficients and all k during the benchmark run (recorded in the results file's run metadata):
+
+| Comparison | Quantity | Max abs. deviation |
+|---|---|--:|
+| Logistic fit vs logistf | Coefficients | 2.2e-16 |
+| Logistic fit vs brglm2 AS_mean | Coefficients | 7.3e-07 |
+| Logistic fit vs brglm2 MPL_Jeffreys | Coefficients | 7.3e-07 |
+| Logistic full vs logistf | Coefficients | 2.2e-16 |
+| Logistic full vs logistf | Profile CI bounds | 5.3e-13 |
+| Logistic full vs logistf | LRT p-values | 2.7e-10 |
+| Logistic numba vs numpy | Coefficients | 2.2e-16 |
+| Cox fit vs coxphf | Coefficients | 8.0e-07 |
+| Cox full vs coxphf | Coefficients | 8.0e-07 |
+| Cox full vs coxphf | Profile CI bounds | 4.3e-08 |
+| Cox full vs coxphf | LRT p-values | 2.5e-08 |
+| Cox numba vs numpy | Coefficients | 3.9e-16 |
+
+---
+
+## Environment
+
+Collected at report-generation time on the benchmark machine.
+
+| Component | Version |
+|---|---|
+| OS | CachyOS |
+| CPU | AMD Ryzen 5 5600X 6-Core Processor |
+| Python | 3.12.13 |
+| firthmodels | 0.8.1 |
+| NumPy / SciPy / Numba | 2.3.5 / 1.16.3 / 0.64.0 |
+| NumPy BLAS | /usr/lib/libopenblas.so.0.3 (openblas 0.3.34, runtime) |
+| R | 4.6.1 |
+| logistf / brglm2 / coxphf | 1.26.1 / 1.1.0 / 1.13.4 |
+| R BLAS | /usr/lib/libopenblas.so.0.3 |
+
+NumPy and R link to the same BLAS library (`/usr/lib/libopenblas.so.0.3`). BLAS threading is left at library defaults for both stacks.
+R packages are compiled from source with `-march=x86-64-v3 -mtune=haswell -O3 -flto=auto`,
+so the R timings are not slowed by a conservative build.
+
+<details>
+<summary>Full R package compile flags (current configuration)</summary>
+
+```
+CFLAGS:  -march=x86-64-v3 -mtune=haswell -O3 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -mpclmul -g1 -ffile-prefix-map=/startdir/src=/usr/src/debug/r -flto=auto -ffat-lto-objects
+FCFLAGS: -O3 -march=x86-64-v3 -mtune=haswell
+```
+
+</details>
+
+---
+
+## Reproducing these results
+
+Requires R with the logistf, brglm2, coxphf, survival, microbenchmark, and
+jsonlite packages installed.
+
+```bash
+# Run both benchmarks (writes results JSON files holding the timings, R
+# reference values, and run metadata), then regenerate the plots and this
+# README
+benchmarks/run_benchmarks.sh
+```

{firthmodels-0.7.2 → firthmodels-0.8.1}/benchmarks/benchmark_cox.py

@@ -15,6 +15,7 @@ import subprocess
 import sys
 import tempfile
 import time
+from datetime import datetime, timezone
 from pathlib import Path
 from typing import Literal
 
@@ -29,7 +30,7 @@ from firthmodels import FirthCoxPH
 N_SAMPLES = 500
 EVENT_RATE = 0.20
 K_VALUES = list(range(5, 35, 5))
-N_RUNS = 15
+N_RUNS = 10
 BASE_SEED = 0
 
 # Solver parameters
@@ -43,8 +44,8 @@ COEF_TOL = 1e-6
 CI_TOL = 1e-6
 PVAL_TOL = 1e-6
 
-# Reduce coxphf benchmark runs for large k
-COXPHF_REDUCE_AFTER = 5
+# Reduce coxphf benchmark runs for large k (0 disables reduction)
+COXPHF_REDUCE_AFTER = 0
 
 
 # -----------------------------------------------------------------------------
@@ -92,7 +93,7 @@ def make_benchmark_data(
     T = -np.log(U) / (baseline_hazard * np.exp(eta))
 
     # Censoring to achieve target event rate
-    censor_quantile = 1 - event_rate
+    censor_quantile = event_rate
     censor_time = np.quantile(T, censor_quantile)
 
     time = np.minimum(T, censor_time)
@@ -279,26 +280,41 @@ def check_agreement(
     r_ci: np.ndarray | None = None,
     py_pval: np.ndarray | None = None,
     r_pval: np.ndarray | None = None,
+    deviations: dict[str, dict[str, float]] | None = None,
 ) -> None:
-    """Check that Python and R results agree within tolerance."""
-    # Check coefficients
+    """Check that Python and R results agree within tolerance.
+
+    If a deviations dict is provided, the observed max absolute differences
+    are merged into it under label, keyed by quantity, taking the max over
+    successive calls (i.e. over k). Recorded before any tolerance failure is
+    raised.
+    """
+    observed: dict[str, float] = {}
+
     coef_diff = np.max(np.abs(py_coef - r_coef))
-    if coef_diff > COEF_TOL:
-        raise AssertionError(f"k={k} ({label}): coef diff {coef_diff:.2e} > {COEF_TOL}")
+    observed["coef"] = float(coef_diff)
 
-    # Check CIs if provided
     if py_ci is not None and r_ci is not None:
-        ci_diff = np.max(np.abs(py_ci - r_ci))
-        if ci_diff > CI_TOL:
-            raise AssertionError(f"k={k} ({label}): CI diff {ci_diff:.2e} > {CI_TOL}")
+        observed["ci"] = float(np.max(np.abs(py_ci - r_ci)))
 
-    # Check p-values if provided
     if py_pval is not None and r_pval is not None:
-        pval_diff = np.max(np.abs(py_pval - r_pval))
-        if pval_diff > PVAL_TOL:
-            raise AssertionError(
-                f"k={k} ({label}): p-value diff {pval_diff:.2e} > {PVAL_TOL}"
-            )
+        observed["pval"] = float(np.max(np.abs(py_pval - r_pval)))
+
+    if deviations is not None:
+        bucket = deviations.setdefault(label, {})
+        for quantity, value in observed.items():
+            bucket[quantity] = max(bucket.get(quantity, 0.0), value)
+
+    if coef_diff > COEF_TOL:
+        raise AssertionError(f"k={k} ({label}): coef diff {coef_diff:.2e} > {COEF_TOL}")
+    if observed.get("ci", 0.0) > CI_TOL:
+        raise AssertionError(
+            f"k={k} ({label}): CI diff {observed['ci']:.2e} > {CI_TOL}"
+        )
+    if observed.get("pval", 0.0) > PVAL_TOL:
+        raise AssertionError(
+            f"k={k} ({label}): p-value diff {observed['pval']:.2e} > {PVAL_TOL}"
+        )
 
 
 # -----------------------------------------------------------------------------
@@ -426,17 +442,17 @@ def compute_min(times: np.ndarray) -> float:
 # Saved results loading
 # -----------------------------------------------------------------------------
 def load_saved_results(
-    saved_csv: str,
+    saved_path: str,
     k_values: list[int],
     load_firthmodels: bool = False,
     load_coxphf: bool = True,
 ) -> tuple[pd.DataFrame, dict[int, dict] | None]:
-    """Load results from a saved CSV file.
+    """Load results from a saved results JSON file.
 
     Parameters
     ----------
-    saved_csv : str
-        Path to CSV with previous results.
+    saved_path : str
+        Path to results JSON with previous results.
     k_values : list of int
         Which k values to load.
     load_firthmodels : bool
@@ -451,7 +467,8 @@ def load_saved_results(
     coxphf_results : dict or None
         coxphf results if load_coxphf=True, else None.
     """
-    saved_df = pd.read_csv(saved_csv)
+    with open(saved_path) as f:
+        saved_df = pd.DataFrame(json.load(f)["results"])
 
     # Determine required columns based on what we're loading
     required_cols = {"k", "n"}
@@ -473,20 +490,21 @@ def load_saved_results(
 
     missing_cols = required_cols - set(saved_df.columns)
     if missing_cols:
-        raise ValueError(f"Saved CSV missing columns: {missing_cols}")
+        raise ValueError(f"Saved results missing columns: {missing_cols}")
 
     saved_n = int(saved_df["n"].iloc[0])
     if saved_n != N_SAMPLES:
         raise ValueError(
-            f"Saved CSV was created with n={saved_n}, but current N_SAMPLES={N_SAMPLES}."
+            f"Saved results were created with n={saved_n}, "
+            f"but current N_SAMPLES={N_SAMPLES}."
         )
 
     saved_k = set(saved_df["k"].tolist())
     missing = set(k_values) - saved_k
     if missing:
-        raise ValueError(f"k values {missing} not in saved CSV")
+        raise ValueError(f"k values {missing} not in saved results")
 
-    # No need to parse JSON - we copy strings directly from saved_df rows
+    # Reference values are copied verbatim from saved_df rows, so no parsing
     return saved_df, None
 
 
@@ -501,8 +519,13 @@ def run_benchmarks(
     saved: str | None = None,
     run_firthmodels: bool = True,
     run_coxphf: bool = True,
-) -> tuple[pd.DataFrame, dict[str, str]]:
-    """Run all benchmarks and return results as DataFrame."""
+) -> tuple[pd.DataFrame, dict[str, str], dict]:
+    """Run all benchmarks and return results as DataFrame.
+
+    Also returns version_info and run_meta (run counts, verification status
+    and observed deviations, versions; saved into the results JSON by
+    main()).
+    """
     version_info = get_python_version_info()
 
     # Load saved results for libraries we're not running
@@ -580,7 +603,7 @@ def run_benchmarks(
         if coxphf_results is None:
             coxphf_results = {}
         if run_coxphf:
-            if coxphf_reduce_after is not None:
+            if coxphf_reduce_after:
                 k_low = [k for k in k_values if k <= coxphf_reduce_after]
                 k_high = [k for k in k_values if k > coxphf_reduce_after]
 
@@ -612,6 +635,7 @@ def run_benchmarks(
             "Verifying results and computing statistics...", file=sys.stderr, flush=True
         )
         results = []
+        deviations: dict[str, dict[str, float]] = {}
 
         for k in k_values:
             X, surv_time, event = make_benchmark_data(
@@ -622,21 +646,32 @@ def run_benchmarks(
             if run_firthmodels and run_coxphf and not skip_verification:
                 py = py_results["numba"][k]
                 r = coxphf_results[k]
-                check_agreement(k, py["fit_coef"], r["fit_coef"], "fit")
+                check_agreement(
+                    k,
+                    py["fit_coef"],
+                    r["fit_coef"],
+                    "fit vs coxphf",
+                    deviations=deviations,
+                )
                 check_agreement(
                     k,
                     py["full_coef"],
                     r["full_coef"],
-                    "full",
+                    "full vs coxphf",
                     py["full_ci"],
                     r["full_ci"],
                     py["full_pval"],
                     r["full_pval"],
+                    deviations=deviations,
                 )
                 # Also check numba vs numpy
                 py_numpy = py_results["numpy"][k]
                 check_agreement(
-                    k, py["fit_coef"], py_numpy["fit_coef"], "numba vs numpy"
+                    k,
+                    py["fit_coef"],
+                    py_numpy["fit_coef"],
+                    "numba vs numpy",
+                    deviations=deviations,
                 )
 
             # Collect timing results
@@ -673,10 +708,10 @@ def run_benchmarks(
                 row["coxphf_fit_ms"] = compute_min(r["fit_times"]) * 1000
                 row["coxphf_full_ms"] = compute_min(r["full_times"]) * 1000
                 # Store for reproducibility
-                row["coxphf_fit_coef"] = json.dumps(r["fit_coef"].tolist())
-                row["coxphf_full_coef"] = json.dumps(r["full_coef"].tolist())
-                row["coxphf_full_ci"] = json.dumps(r["full_ci"].tolist())
-                row["coxphf_full_pval"] = json.dumps(r["full_pval"].tolist())
+                row["coxphf_fit_coef"] = np.asarray(r["fit_coef"]).tolist()
+                row["coxphf_full_coef"] = np.asarray(r["full_coef"]).tolist()
+                row["coxphf_full_ci"] = np.asarray(r["full_ci"]).tolist()
+                row["coxphf_full_pval"] = np.asarray(r["full_pval"]).tolist()
             else:
                 assert saved_row is not None
                 row["coxphf_fit_ms"] = saved_row["coxphf_fit_ms"]
@@ -688,7 +723,39 @@ def run_benchmarks(
 
             results.append(row)
 
-    return pd.DataFrame(results), version_info
+    reduced = n_runs if n_runs <= 3 else max(3, n_runs // 3)
+    run_meta = {
+        "schema_version": 1,
+        "model": "cox",
+        "created_utc": datetime.now(timezone.utc).isoformat(timespec="seconds"),
+        "n_runs": n_runs,
+        "r_reduced_runs": (
+            {
+                "libraries": ["coxphf"],
+                "for_k_above": coxphf_reduce_after,
+                "n_runs": reduced,
+            }
+            if coxphf_reduce_after
+            else None
+        ),
+        "k_values": list(k_values),
+        "libraries_run": [
+            name
+            for name, flag in [
+                ("firthmodels", run_firthmodels),
+                ("coxphf", run_coxphf),
+            ]
+            if flag
+        ],
+        "loaded_from_saved": saved,
+        "verification": {
+            "performed": bool(deviations),
+            "tolerances": {"coef": COEF_TOL, "ci": CI_TOL, "pval": PVAL_TOL},
+            "max_abs_deviations": deviations,
+        },
+        "versions": version_info,
+    }
+    return pd.DataFrame(results), version_info, run_meta
 
 
 def print_table(df: pd.DataFrame) -> None:
@@ -743,14 +810,14 @@ def main():
         "--out",
         type=str,
         default=None,
-        help="Save results to CSV file",
+        help="Save results (timings, R references, run metadata) to JSON file",
     )
     parser.add_argument(
         "--saved",
         type=str,
         default=None,
-        metavar="CSV",
-        help="Load non-selected libraries from this CSV.",
+        metavar="JSON",
+        help="Load non-selected libraries from this results JSON.",
     )
     parser.add_argument(
         "--firthmodels",
@@ -807,7 +874,7 @@ def main():
         )
 
     # Run benchmarks
-    df, version_info = run_benchmarks(
+    df, version_info, run_meta = run_benchmarks(
         k_values=k_values,
         n_runs=args.n_runs,
         skip_verification=args.skip_verification,
@@ -821,7 +888,11 @@ def main():
     print_table(df)
 
     if args.out:
-        df.to_csv(args.out, index=False)
+        with open(args.out, "w") as f:
+            json.dump(
+                {"meta": run_meta, "results": df.to_dict(orient="records")}, f, indent=2
+            )
+            f.write("\n")
         print(f"\nResults saved to {args.out}", file=sys.stderr)