evalrx 0.1.2__py3-none-any.whl

evalrx/agent_assets/skills/outcome-driver-analysis/references/model_selection.md

@@ -0,0 +1,72 @@
+# Model selection reference
+
+Dense decision tables supporting `SKILL.md` steps 3-6. Read the relevant table when
+you reach that step; don't load this whole file into a report.
+
+## Step 3 -- univariate test vs. outcome, by variable type
+
+| Variable type | Assumptions met | Primary test | Assumptions violated / robust fallback |
+|---|---|---|---|
+| Categorical (vs. binary outcome) | all expected cell counts >= 5 | Chi-square test of independence | Fisher's exact test (small/sparse cells); Cochran-Mantel-Haenszel test if stratifying by a third categorical variable |
+| Continuous (vs. binary outcome) | approx. normal within each group, similar variances | Two-sample t-test (Student's) | Welch's t-test (unequal variances); Mann-Whitney U / Wilcoxon rank-sum (non-normal, robust default); permutation test (small n, no distributional assumption) |
+| Ordinal (vs. binary outcome) | -- | Mann-Whitney U / Wilcoxon rank-sum | Cumulative-link / ordinal logistic model if treating ordinal as a predictor in a larger model |
+
+Effect sizes to report alongside each test: Cramer's V or odds ratio (categorical),
+Cohen's d or rank-biserial correlation (continuous/ordinal).
+
+R: `chisq.test()`, `fisher.test()`, `t.test()`, `wilcox.test()`.
+Python: `scipy.stats.chi2_contingency`, `scipy.stats.fisher_exact`, `scipy.stats.ttest_ind`,
+`scipy.stats.mannwhitneyu`.
+
+## Step 4 -- marginal screening
+
+| Screening approach | When to use | R | Python |
+|---|---|---|---|
+| Univariate logistic regression per variable, rank by p-value | Default screening method | `glm(outcome ~ x, family = binomial)` | `statsmodels.api.Logit` |
+| AIC improvement over null model | Comparing variables on a consistent scale, useful when p-values are similar across many variables | `AIC(model_null) - AIC(model_x)` | `model_x.aic` vs. null model `aic` |
+| Reuse step-3 bivariate test p-values | Quick pass when there are few candidate variables | -- | -- |
+
+Screening narrows the candidate list; it does not replace fitting the full
+multivariable model in step 5. Keep a variable in the candidate set even if its
+marginal signal is weak when subject-matter reasoning or the step-3 conditioning
+check suggests a real, confounded effect.
+
+## Step 5 -- outcome type and clustering -> model family
+
+| Outcome type | Observations independent? | Model | R | Python |
+|---|---|---|---|---|
+| Binary (success/fail) | Yes | Logistic regression (binomial GLM) | `glm(y ~ ., family = binomial)` | `statsmodels.api.Logit` / `sklearn.linear_model.LogisticRegression` (use `statsmodels` for inference/p-values) |
+| Binary (success/fail) | No -- clustered/repeated (e.g. multiple examples per model/prompt/subject) | Mixed-effects logistic regression (GLMM) | `lme4::glmer(y ~ . + (1\|cluster), family = binomial)`, or `glmmTMB` | `statsmodels.genmod.bayes_mixed_glm.BinomialBayesMixedGLM`, or `pymer4` (R-backed) |
+| Continuous | Yes | Linear regression | `lm(y ~ .)` | `statsmodels.api.OLS` |
+| Continuous | No -- clustered/repeated | Linear mixed-effects model (LMM) | `lme4::lmer(y ~ . + (1\|cluster))` | `statsmodels.regression.mixed_linear_model.MixedLM` |
+| Count | Yes | Poisson or negative binomial regression (check for overdispersion) | `glm(y ~ ., family = poisson)`, `MASS::glm.nb()` | `statsmodels.discrete.discrete_model.Poisson`, `.NegativeBinomial` |
+| Count | No -- clustered/repeated | Mixed-effects Poisson/negative binomial | `glmmTMB(y ~ . + (1\|cluster), family = nbinom2)` | `statsmodels` mixed count models are limited -- R is the more mature option here |
+
+Why not linear regression for a binary outcome: predictions can fall outside [0, 1],
+and the constant-variance / normal-error assumptions behind linear regression's
+inference (t-tests, F-tests, prediction intervals) don't hold for a 0/1 target.
+Logistic regression models the log-odds of success directly and its errors match a
+Bernoulli outcome.
+
+When to add a random effect (mixed-effects model): observations share a grouping
+structure that could induce correlated errors -- e.g. several examples generated by
+the same underlying model, drawn from the same prompt template, or measured on the
+same subject. Ignoring this structure understates standard errors and can produce
+false-positive findings. A rule of thumb: if you can name the grouping variable and
+each group contributes more than one row, consider a random intercept for it; add a
+random slope only if there's reason to think the effect of a predictor itself varies
+by group and there's enough data per group to estimate it.
+
+## Step 6 -- goodness-of-fit and diagnostics, by model
+
+| Check | Applies to | R | Python |
+|---|---|---|---|
+| Hosmer-Lemeshow test | Logistic regression (few continuous predictors) | `ResourceSelection::hoslem.test()` | `statsmodels` has no built-in; implement via binned deciles of predicted probability |
+| Deviance / Pearson residuals | Any GLM | `residuals(model, type = "deviance")` | `model.resid_deviance`, `model.resid_pearson` |
+| Binned residual plot | Logistic regression / GLMM | `arm::binnedplot()` | plot residuals vs. fitted, binned manually |
+| VIF (multicollinearity) | Any regression with >1 predictor | `car::vif(model)` | `statsmodels.stats.outliers_influence.variance_inflation_factor` |
+| Influence diagnostics | Any GLM | `influence.measures(model)` | `model.get_influence()` |
+| ROC / AUC | Binary outcome model | `pROC::roc()` | `sklearn.metrics.roc_curve`, `roc_auc_score` |
+| Calibration plot | Binary outcome model | `rms::val.prob()` | manual: bin predicted probability, plot observed vs. predicted rate |
+| Random-effect variance / ICC | GLMM/LMM | `performance::icc(model)` | `pymer4` wraps R; native Python support is limited |
+| Convergence check | GLMM/LMM | inspect `model@optinfo$conv` | inspect optimizer convergence flag on the fit result |

evalrx/agent_assets/skills/outcome-driver-analysis/scripts/explanatory_var_eda.R

@@ -0,0 +1,130 @@
+# Step 2 starting template: explanatory-variable EDA.
+#
+# Given a dataframe and a list of explanatory variable columns, produce:
+# - a per-variable distribution plot (histogram for continuous, bar chart for categorical)
+# - an outlier / missingness summary table
+# - a mixed-type correlation / association matrix among the explanatory variables
+#
+# This is a template to adapt to the real column names and data -- not a black box.
+#
+# Usage:
+#   Rscript explanatory_var_eda.R data.csv "age,region,score" output/
+
+suppressMessages({
+  library(dplyr)
+  library(ggplot2)
+})
+
+infer_variable_types <- function(df, variables) {
+  sapply(variables, function(v) {
+    if (is.numeric(df[[v]]) && length(unique(df[[v]])) > 10) "continuous" else "categorical"
+  })
+}
+
+outlier_summary <- function(df, var) {
+  q <- quantile(df[[var]], c(0.25, 0.75), na.rm = TRUE)
+  iqr <- q[2] - q[1]
+  lower <- q[1] - 1.5 * iqr
+  upper <- q[2] + 1.5 * iqr
+  n_outliers <- sum(df[[var]] < lower | df[[var]] > upper, na.rm = TRUE)
+  data.frame(variable = var, n_outliers = n_outliers, lower_bound = lower, upper_bound = upper)
+}
+
+rare_category_summary <- function(df, var, threshold = 0.01) {
+  counts <- prop.table(table(df[[var]]))
+  rare <- names(counts[counts < threshold])
+  data.frame(variable = var, n_categories = length(counts), rare_categories = paste(rare, collapse = ";"))
+}
+
+missingness_summary <- function(df, variables) {
+  do.call(rbind, lapply(variables, function(v) {
+    n_missing <- sum(is.na(df[[v]]))
+    data.frame(variable = v, n_missing = n_missing, pct_missing = round(100 * n_missing / nrow(df), 1))
+  }))
+}
+
+cramers_v <- function(x, y) {
+  ct <- table(x, y)
+  chi2 <- suppressWarnings(chisq.test(ct)$statistic)
+  n <- sum(ct)
+  phi2 <- chi2 / n
+  r <- nrow(ct); k <- ncol(ct)
+  phi2_corr <- max(0, phi2 - (k - 1) * (r - 1) / (n - 1))
+  r_corr <- r - (r - 1)^2 / (n - 1)
+  k_corr <- k - (k - 1)^2 / (n - 1)
+  sqrt(phi2_corr / min(k_corr - 1, r_corr - 1))
+}
+
+correlation_ratio <- function(categorical, continuous) {
+  d <- na.omit(data.frame(cat = categorical, cont = continuous))
+  overall_mean <- mean(d$cont)
+  agg <- aggregate(cont ~ cat, d, function(g) c(n = length(g), mean = mean(g)))
+  ss_between <- sum(sapply(split(d$cont, d$cat), function(g) length(g) * (mean(g) - overall_mean)^2))
+  ss_total <- sum((d$cont - overall_mean)^2)
+  if (ss_total > 0) sqrt(ss_between / ss_total) else 0
+}
+
+association_matrix <- function(df, var_types) {
+  variables <- names(var_types)
+  mat <- matrix(NA, length(variables), length(variables), dimnames = list(variables, variables))
+  for (a in variables) {
+    for (b in variables) {
+      if (a == b) { mat[a, b] <- 1; next }
+      ta <- var_types[[a]]; tb <- var_types[[b]]
+      if (ta == "continuous" && tb == "continuous") {
+        mat[a, b] <- cor(df[[a]], df[[b]], use = "pairwise.complete.obs")
+      } else if (ta == "categorical" && tb == "categorical") {
+        mat[a, b] <- cramers_v(df[[a]], df[[b]])
+      } else {
+        if (ta == "categorical") mat[a, b] <- correlation_ratio(df[[a]], df[[b]])
+        else mat[a, b] <- correlation_ratio(df[[b]], df[[a]])
+      }
+    }
+  }
+  mat
+}
+
+plot_distribution <- function(df, var, var_type, out_dir) {
+  p <- if (var_type == "continuous") {
+    ggplot(df, aes(.data[[var]])) + geom_histogram(bins = 30, fill = "steelblue", color = "white")
+  } else {
+    ggplot(df, aes(.data[[var]])) + geom_bar(fill = "steelblue")
+  }
+  p <- p + labs(title = var, x = NULL) + theme_minimal() +
+    theme(panel.grid.minor = element_blank())
+  ggsave(file.path(out_dir, paste0("dist_", var, ".pdf")), p, width = 5, height = 3.5)
+}
+
+run <- function(csv_path, variables, out_dir) {
+  dir.create(out_dir, showWarnings = FALSE, recursive = TRUE)
+  df <- read.csv(csv_path)
+  var_types <- infer_variable_types(df, variables)
+
+  for (v in variables) plot_distribution(df, v, var_types[[v]], out_dir)
+
+  cont_vars <- variables[var_types == "continuous"]
+  cat_vars <- variables[var_types == "categorical"]
+  outliers <- do.call(rbind, lapply(cont_vars, outlier_summary, df = df))
+  rares <- do.call(rbind, lapply(cat_vars, rare_category_summary, df = df))
+  missing <- missingness_summary(df, variables)
+  assoc <- association_matrix(df, var_types)
+
+  write.csv(outliers, file.path(out_dir, "outlier_summary.csv"), row.names = FALSE)
+  write.csv(rares, file.path(out_dir, "rare_category_summary.csv"), row.names = FALSE)
+  write.csv(missing, file.path(out_dir, "missingness_summary.csv"), row.names = FALSE)
+  write.csv(assoc, file.path(out_dir, "association_matrix.csv"))
+
+  cat("Variable types:\n"); print(var_types)
+  cat("\nMissingness:\n"); print(missing)
+  cat("\nOutliers (continuous):\n"); print(outliers)
+  cat("\nRare categories (categorical):\n"); print(rares)
+  cat("\nAssociation matrix:\n"); print(round(assoc, 2))
+}
+
+if (sys.nframe() == 0) {
+  args <- commandArgs(trailingOnly = TRUE)
+  csv_path <- args[1]
+  variables <- strsplit(args[2], ",")[[1]]
+  out_dir <- ifelse(length(args) >= 3, args[3], "output/figures")
+  run(csv_path, variables, out_dir)
+}

evalrx/agent_assets/skills/outcome-driver-analysis/scripts/explanatory_var_eda.py

@@ -0,0 +1,150 @@
+"""
+Step 2 starting template: explanatory-variable EDA.
+
+Given a dataframe and a list of explanatory variable columns, produce:
+- a per-variable distribution plot (histogram for continuous, bar chart for categorical)
+- an outlier / missingness summary table
+- a mixed-type correlation / association matrix among the explanatory variables
+
+This is a template to adapt to the real column names and data -- not a black box.
+Edit CONTINUOUS_VARS / CATEGORICAL_VARS below, or pass them in when calling the
+functions from another script / notebook.
+
+Usage:
+    python explanatory_var_eda.py data.csv --vars age,region,score --out output/
+"""
+
+import argparse
+import itertools
+import os
+
+import matplotlib.pyplot as plt
+import numpy as np
+import pandas as pd
+
+
+def infer_variable_types(df: pd.DataFrame, variables: list[str]) -> dict[str, str]:
+    """Heuristic: numeric dtype with many unique values -> continuous, else categorical."""
+    types = {}
+    for v in variables:
+        if pd.api.types.is_numeric_dtype(df[v]) and df[v].nunique() > 10:
+            types[v] = "continuous"
+        else:
+            types[v] = "categorical"
+    return types
+
+
+def outlier_summary(df: pd.DataFrame, var: str) -> dict:
+    """IQR rule for continuous variables."""
+    q1, q3 = df[var].quantile([0.25, 0.75])
+    iqr = q3 - q1
+    lower, upper = q1 - 1.5 * iqr, q3 + 1.5 * iqr
+    n_outliers = ((df[var] < lower) | (df[var] > upper)).sum()
+    return {"variable": var, "n_outliers": int(n_outliers), "lower_bound": lower, "upper_bound": upper}
+
+
+def rare_category_summary(df: pd.DataFrame, var: str, threshold: float = 0.01) -> dict:
+    """Flag categories making up less than `threshold` fraction of non-missing rows."""
+    counts = df[var].value_counts(normalize=True)
+    rare = counts[counts < threshold]
+    return {"variable": var, "n_categories": len(counts), "rare_categories": list(rare.index)}
+
+
+def missingness_summary(df: pd.DataFrame, variables: list[str]) -> pd.DataFrame:
+    rows = []
+    for v in variables:
+        n_missing = df[v].isna().sum()
+        rows.append({"variable": v, "n_missing": int(n_missing), "pct_missing": round(100 * n_missing / len(df), 1)})
+    return pd.DataFrame(rows)
+
+
+def cramers_v(x: pd.Series, y: pd.Series) -> float:
+    """Association between two categorical variables, bias-corrected."""
+    from scipy.stats import chi2_contingency
+
+    ct = pd.crosstab(x, y)
+    chi2 = chi2_contingency(ct)[0]
+    n = ct.sum().sum()
+    phi2 = chi2 / n
+    r, k = ct.shape
+    phi2_corr = max(0, phi2 - (k - 1) * (r - 1) / (n - 1))
+    r_corr = r - (r - 1) ** 2 / (n - 1)
+    k_corr = k - (k - 1) ** 2 / (n - 1)
+    return float(np.sqrt(phi2_corr / min(k_corr - 1, r_corr - 1)))
+
+
+def correlation_ratio(categorical: pd.Series, continuous: pd.Series) -> float:
+    """Eta: association between a categorical and a continuous variable."""
+    df = pd.DataFrame({"cat": categorical, "cont": continuous}).dropna()
+    overall_mean = df["cont"].mean()
+    groups = df.groupby("cat")["cont"]
+    ss_between = sum(len(g) * (g.mean() - overall_mean) ** 2 for _, g in groups)
+    ss_total = ((df["cont"] - overall_mean) ** 2).sum()
+    return float(np.sqrt(ss_between / ss_total)) if ss_total > 0 else 0.0
+
+
+def association_matrix(df: pd.DataFrame, var_types: dict[str, str]) -> pd.DataFrame:
+    """Mixed-type association matrix: Pearson r (cont-cont), Cramer's V (cat-cat), eta (mixed)."""
+    variables = list(var_types.keys())
+    mat = pd.DataFrame(index=variables, columns=variables, dtype=float)
+    for a, b in itertools.combinations_with_replacement(variables, 2):
+        if a == b:
+            mat.loc[a, b] = 1.0
+            continue
+        ta, tb = var_types[a], var_types[b]
+        if ta == "continuous" and tb == "continuous":
+            val = df[[a, b]].corr().iloc[0, 1]
+        elif ta == "categorical" and tb == "categorical":
+            val = cramers_v(df[a], df[b])
+        else:
+            cat_var, cont_var = (a, b) if ta == "categorical" else (b, a)
+            val = correlation_ratio(df[cat_var], df[cont_var])
+        mat.loc[a, b] = mat.loc[b, a] = val
+    return mat
+
+
+def plot_distribution(df: pd.DataFrame, var: str, var_type: str, out_dir: str) -> None:
+    fig, ax = plt.subplots(figsize=(5, 3.5))
+    if var_type == "continuous":
+        ax.hist(df[var].dropna(), bins=30, color="steelblue", edgecolor="white")
+    else:
+        df[var].value_counts().plot(kind="bar", ax=ax, color="steelblue")
+    ax.set_title(var)
+    ax.spines[["top", "right"]].set_visible(False)
+    fig.tight_layout()
+    fig.savefig(os.path.join(out_dir, f"dist_{var}.pdf"))
+    plt.close(fig)
+
+
+def run(csv_path: str, variables: list[str], out_dir: str) -> None:
+    os.makedirs(out_dir, exist_ok=True)
+    df = pd.read_csv(csv_path)
+    var_types = infer_variable_types(df, variables)
+
+    for v in variables:
+        plot_distribution(df, v, var_types[v], out_dir)
+
+    outliers = [outlier_summary(df, v) for v, t in var_types.items() if t == "continuous"]
+    rares = [rare_category_summary(df, v) for v, t in var_types.items() if t == "categorical"]
+    missing = missingness_summary(df, variables)
+    assoc = association_matrix(df, var_types)
+
+    pd.DataFrame(outliers).to_csv(os.path.join(out_dir, "outlier_summary.csv"), index=False)
+    pd.DataFrame(rares).to_csv(os.path.join(out_dir, "rare_category_summary.csv"), index=False)
+    missing.to_csv(os.path.join(out_dir, "missingness_summary.csv"), index=False)
+    assoc.to_csv(os.path.join(out_dir, "association_matrix.csv"))
+
+    print("Variable types:", var_types)
+    print("\nMissingness:\n", missing)
+    print("\nOutliers (continuous):\n", pd.DataFrame(outliers))
+    print("\nRare categories (categorical):\n", pd.DataFrame(rares))
+    print("\nAssociation matrix:\n", assoc.round(2))
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("csv_path")
+    parser.add_argument("--vars", required=True, help="comma-separated explanatory variable column names")
+    parser.add_argument("--out", default="output/figures")
+    args = parser.parse_args()
+    run(args.csv_path, args.vars.split(","), args.out)

evalrx/agent_assets/skills/outcome-driver-analysis/scripts/fit_outcome_model.R

@@ -0,0 +1,181 @@
+# Steps 5-7 starting template: fit a justified model for a binary outcome, test
+# per-variable significance, run goodness-of-fit diagnostics, and visualize results.
+#
+# Given a dataframe, a binary outcome column, a list of predictor columns, and an
+# optional cluster column (for repeated/clustered observations):
+# - fits a plain logistic regression (glm, family = binomial) when there's no clustering
+# - fits a mixed-effects logistic regression (lme4::glmer, random intercept for the
+#   cluster) when clustering is present -- this is the language where GLMMs are most
+#   mature, per references/model_selection.md
+# - reports per-variable Wald and likelihood-ratio significance with odds ratios
+# - runs Hosmer-Lemeshow (fixed-effects case), VIF, ROC/AUC, and a calibration check,
+#   plus ICC/convergence checks for the mixed-effects case
+# - produces a coefficient/odds-ratio plot, an ROC curve, and a calibration plot
+#
+# This is a template to adapt to the real column names and data -- not a black box.
+#
+# Usage:
+#   Rscript fit_outcome_model.R data.csv outcome "age,region,score" output/ [cluster]
+
+suppressMessages({
+  library(dplyr)
+  library(ggplot2)
+  library(car)      # vif()
+  library(pROC)     # roc()
+})
+
+fit_model <- function(df, outcome, predictors, cluster = NULL) {
+  formula_str <- paste(outcome, "~", paste(predictors, collapse = " + "))
+  if (is.null(cluster)) {
+    message("Fit plain logistic regression (observations treated as independent).")
+    glm(as.formula(formula_str), data = df, family = binomial)
+  } else {
+    suppressMessages(library(lme4))
+    message(sprintf("Observations are clustered by '%s': fit mixed-effects logistic regression",
+                     cluster))
+    mixed_formula <- paste(formula_str, "+ (1 |", cluster, ")")
+    lme4::glmer(as.formula(mixed_formula), data = df, family = binomial,
+                control = lme4::glmerControl(optimizer = "bobyqa"))
+  }
+}
+
+is_mixed <- function(model) inherits(model, "merMod")
+
+per_variable_significance <- function(df, outcome, predictors, full_model, cluster = NULL) {
+  # a categorical predictor expands to one coefficient per non-reference level (e.g.
+  # "regionnorth", "regionsouth") -- report one row per level, but share a single
+  # likelihood-ratio p-value (a joint test dropping the whole variable) across them
+  formula_base <- if (is_mixed(full_model)) paste("+ (1 |", cluster, ")") else ""
+  coef_table <- summary(full_model)$coefficients
+  pr_col <- grep("Pr", colnames(coef_table), value = TRUE)[1]
+  rows <- lapply(predictors, function(v) {
+    reduced_predictors <- setdiff(predictors, v)
+    reduced_rhs <- if (length(reduced_predictors) > 0) paste(reduced_predictors, collapse = " + ") else "1"
+    reduced_formula <- paste(outcome, "~", reduced_rhs, formula_base)
+    reduced_model <- if (is_mixed(full_model)) {
+      lme4::glmer(as.formula(reduced_formula), data = df, family = binomial)
+    } else {
+      glm(as.formula(reduced_formula), data = df, family = binomial)
+    }
+    lrt <- anova(reduced_model, full_model, test = "Chisq")
+    # stats::anova.glm names this column "Pr(>Chi)"; lme4:::anova.merMod names it
+    # "Pr(>Chisq)" -- look it up rather than hardcoding either spelling
+    lrt_p <- lrt[[grep("^Pr", colnames(lrt), value = TRUE)]][2]
+
+    coef_names <- grep(paste0("^", v), names(fixef_or_coef(full_model)), value = TRUE)
+    do.call(rbind, lapply(coef_names, function(coef_name) {
+      est <- fixef_or_coef(full_model)[[coef_name]]
+      se <- coef_table[coef_name, "Std. Error"]
+      wald_p <- coef_table[coef_name, pr_col]
+      data.frame(
+        variable = coef_name,
+        odds_ratio = exp(est),
+        ci_low = exp(est - 1.96 * se),
+        ci_high = exp(est + 1.96 * se),
+        wald_p = wald_p,
+        lrt_p = lrt_p
+      )
+    }))
+  })
+  do.call(rbind, rows)
+}
+
+fixef_or_coef <- function(model) if (is_mixed(model)) lme4::fixef(model) else coef(model)
+
+hosmer_lemeshow <- function(y_true, y_pred, n_bins = 10) {
+  bins <- cut(y_pred, breaks = quantile(y_pred, probs = seq(0, 1, length.out = n_bins + 1)),
+              include.lowest = TRUE)
+  obs <- tapply(y_true, bins, sum)
+  exp <- tapply(y_pred, bins, sum)
+  n <- tapply(y_true, bins, length)
+  stat <- sum((obs - exp)^2 / (exp * (1 - exp / n)), na.rm = TRUE)
+  p_value <- pchisq(stat, df = n_bins - 2, lower.tail = FALSE)
+  list(statistic = stat, p_value = p_value)
+}
+
+plot_coefficients <- function(sig_table, out_dir) {
+  p <- ggplot(sig_table, aes(x = odds_ratio, y = reorder(variable, odds_ratio))) +
+    geom_point(color = "steelblue") +
+    geom_errorbarh(aes(xmin = ci_low, xmax = ci_high), height = 0.2, color = "steelblue") +
+    geom_vline(xintercept = 1, linetype = "dashed", color = "gray50") +
+    labs(x = "Odds ratio (95% CI)", y = NULL) +
+    theme_minimal()
+  ggsave(file.path(out_dir, "coefficient_plot.pdf"), p, width = 5, height = 0.5 * nrow(sig_table) + 1)
+}
+
+plot_roc <- function(y_true, y_pred, out_dir) {
+  roc_obj <- pROC::roc(y_true, y_pred, quiet = TRUE)
+  pdf(file.path(out_dir, "roc_curve.pdf"), width = 4, height = 4)
+  plot(roc_obj, col = "steelblue", main = sprintf("AUC = %.3f", pROC::auc(roc_obj)))
+  dev.off()
+  as.numeric(pROC::auc(roc_obj))
+}
+
+plot_calibration <- function(y_true, y_pred, out_dir, n_bins = 10) {
+  bins <- cut(y_pred, breaks = quantile(y_pred, probs = seq(0, 1, length.out = n_bins + 1)),
+              include.lowest = TRUE)
+  observed <- tapply(y_true, bins, mean)
+  predicted <- tapply(y_pred, bins, mean)
+  df <- data.frame(observed = observed, predicted = predicted)
+  p <- ggplot(df, aes(predicted, observed)) +
+    geom_point(color = "steelblue") + geom_line(color = "steelblue") +
+    geom_abline(slope = 1, intercept = 0, linetype = "dashed", color = "gray50") +
+    labs(x = "Mean predicted probability", y = "Observed proportion") +
+    theme_minimal()
+  ggsave(file.path(out_dir, "calibration_plot.pdf"), p, width = 4, height = 4)
+}
+
+run <- function(csv_path, outcome, predictors, out_dir, cluster = NULL) {
+  dir.create(out_dir, showWarnings = FALSE, recursive = TRUE)
+  df <- read.csv(csv_path)
+  df <- df[complete.cases(df[, c(outcome, predictors)]), ]
+
+  model <- fit_model(df, outcome, predictors, cluster)
+  print(summary(model))
+
+  sig_table <- per_variable_significance(df, outcome, predictors, model, cluster)
+  write.csv(sig_table, file.path(out_dir, "per_variable_significance.csv"), row.names = FALSE)
+  cat("\nPer-variable significance (adjusted for other predictors):\n"); print(sig_table)
+
+  y_pred <- predict(model, type = "response")
+
+  if (is_mixed(model)) {
+    cat("\nMixed-effects model: checking random-effect variance, ICC, and convergence.\n")
+    print(lme4::VarCorr(model))
+    if (!is.null(model@optinfo$conv$lme4$messages)) {
+      cat("Convergence warnings:\n"); print(model@optinfo$conv$lme4$messages)
+    } else {
+      cat("No convergence warnings.\n")
+    }
+  } else {
+    hl <- hosmer_lemeshow(df[[outcome]], y_pred)
+    cat(sprintf("\nHosmer-Lemeshow: chi2=%.3f, p=%.4g\n", hl$statistic, hl$p_value))
+    vif_vals <- car::vif(model)
+    # car::vif() returns a plain named vector when every predictor has 1 df, but a
+    # GVIF/Df/GVIF^(1/(2*Df)) matrix as soon as any factor predictor has >2 levels --
+    # use the adjusted GVIF^(1/(2*Df))^2 (comparable to a plain VIF) in that case
+    vif_df <- if (is.matrix(vif_vals)) {
+      data.frame(variable = rownames(vif_vals), vif = vif_vals[, "GVIF^(1/(2*Df))"]^2)
+    } else {
+      data.frame(variable = names(vif_vals), vif = vif_vals)
+    }
+    write.csv(vif_df, file.path(out_dir, "vif.csv"), row.names = FALSE)
+    cat("\nVIF:\n"); print(vif_df)
+  }
+
+  auc <- plot_roc(df[[outcome]], y_pred, out_dir)
+  plot_calibration(df[[outcome]], y_pred, out_dir)
+  plot_coefficients(sig_table, out_dir)
+  cat(sprintf("\nAUC: %.3f\n", auc))
+  cat(sprintf("Plots written to %s: coefficient_plot.pdf, roc_curve.pdf, calibration_plot.pdf\n", out_dir))
+}
+
+if (sys.nframe() == 0) {
+  args <- commandArgs(trailingOnly = TRUE)
+  csv_path <- args[1]
+  outcome <- args[2]
+  predictors <- strsplit(args[3], ",")[[1]]
+  out_dir <- ifelse(length(args) >= 4, args[4], "output")
+  cluster <- if (length(args) >= 5) args[5] else NULL
+  run(csv_path, outcome, predictors, out_dir, cluster)
+}