ssme-lite 0.1.0__tar.gz

ssme_lite-0.1.0/CHANGELOG.md

@@ -0,0 +1,9 @@
+# Changelog
+
+## 0.1.0
+
+- Public estimator, prediction-matrix, split, and report interfaces.
+- Accuracy, ROC AUC, average precision, and calibration-error estimates.
+- Offline interactive reports and CSV/JSON exports.
+- Optional controlled leave-one-model-out influence diagnostics.
+- English documentation, runnable quickstart, real-data tutorials, and release validation.

ssme_lite-0.1.0/CONTRIBUTING.md

@@ -0,0 +1,14 @@
+# Contributing
+
+Use Python 3.10 or newer and run commands from the repository root.
+
+```bash
+python -m pip install -e '.[dev,docs]'
+python -m pytest -q
+python examples/quickstart.py
+mkdocs build --strict
+```
+
+Keep the public imports in `ssme_lite` stable. Document new parameters and update runnable examples when behavior changes. Use small synthetic inputs for tests; keep heavyweight datasets and model downloads out of the default test suite. The optional official-reference audit skips when its external dependencies or reference tree are unavailable.
+
+Describe the behavior change and validation in pull requests. Do not commit datasets, model weights, credentials, generated distribution files, or local environments. See `docs/releasing.md` for release verification.

ssme_lite-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 jason
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

ssme_lite-0.1.0/MANIFEST.in

@@ -0,0 +1,11 @@
+include LICENSE README.md CHANGELOG.md CONTRIBUTING.md mkdocs.yml
+recursive-include docs *.md
+recursive-include examples *.py
+recursive-include tests *.py
+recursive-include ssme_lite *.py py.typed *.html *.css *.js
+prune experiments
+prune data
+prune results
+prune site
+prune .github
+global-exclude __pycache__ *.py[cod] .DS_Store

ssme_lite-0.1.0/PKG-INFO

@@ -0,0 +1,120 @@
+Metadata-Version: 2.4
+Name: ssme-lite
+Version: 0.1.0
+Summary: Semi-supervised evaluation and selection of existing classifiers
+Author: Junje Shen
+License-Expression: MIT
+Project-URL: Documentation, https://github.com/JasonShen2002/ssme-lite/blob/main/docs/index.md
+Project-URL: Source, https://github.com/JasonShen2002/ssme-lite
+Project-URL: Issues, https://github.com/JasonShen2002/ssme-lite/issues
+Project-URL: Changelog, https://github.com/JasonShen2002/ssme-lite/blob/main/CHANGELOG.md
+Keywords: semi-supervised,model-evaluation,model-selection,machine-learning
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.24
+Requires-Dist: scipy>=1.10
+Requires-Dist: pandas>=2
+Requires-Dist: scikit-learn>=1.3
+Requires-Dist: matplotlib>=3.7
+Requires-Dist: joblib>=1.3
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Provides-Extra: docs
+Requires-Dist: mkdocs<2,>=1.6; extra == "docs"
+Provides-Extra: official
+Requires-Dist: statsmodels>=0.14; extra == "official"
+Provides-Extra: transfer
+Requires-Dist: torch>=2; extra == "transfer"
+Requires-Dist: torchvision>=0.15; extra == "transfer"
+Requires-Dist: pillow>=9; extra == "transfer"
+Dynamic: license-file
+
+# SSME-Lite
+
+**Evaluate existing classifiers with a few labels and many unlabeled predictions.**
+
+SSME-Lite packages semi-supervised model evaluation into a small Python interface: provide model probabilities and partial labels, estimate performance, rank candidates, and export an interactive HTML report that opens offline.
+
+- **One estimator:** `fit()` followed by `report()`.
+- **Four metrics:** Accuracy, ROC AUC, average precision (`auprc`), and ECE.
+- **Flexible inputs:** precomputed probabilities or already fitted sklearn-style models.
+- **Portable outputs:** HTML, CSV, JSON, and metric plots.
+- **Reproducible experiments:** explicit seeds, shared splits, and optional model-influence diagnostics.
+- **Lightweight core:** no PyTorch dependency; transfer learning is optional.
+
+## Installation
+
+Python 3.10 or newer:
+
+```bash
+python -m pip install ssme-lite
+```
+
+To work from a checkout:
+
+```bash
+git clone https://github.com/JasonShen2002/ssme-lite.git
+cd ssme-lite
+python -m pip install -e '.[dev]'
+```
+
+## Minimal interface
+
+```python
+from ssme_lite import SSMEEstimator
+
+# scores: (samples, models), positive-class probabilities for binary tasks.
+# y_partial: one integer label per row; -1 means unknown.
+estimator = SSMEEstimator(random_state=0)
+estimator.fit(scores, y_partial, model_names=model_names)
+report = estimator.report(primary_metric="accuracy")
+print(report.ranking("accuracy"))
+report.save("results/my_report")
+```
+
+Every class must have at least one observed label. For multiclass tasks, use `(samples, models, classes)` probabilities and class indices `0..K-1`.
+
+For a complete example without external data, run `python examples/quickstart.py` from the checkout. It generates a small synthetic dataset, fits SSME, and saves `results/quickstart/report.html`. This example illustrates the interface, not an empirical performance claim.
+
+## Documentation and tutorials
+
+- [Documentation home](https://github.com/JasonShen2002/ssme-lite/blob/main/docs/index.md)
+- [Five-minute quickstart](https://github.com/JasonShen2002/ssme-lite/blob/main/docs/quickstart.md)
+- [Use your own models](https://github.com/JasonShen2002/ssme-lite/blob/main/docs/guides/inputs.md)
+- [Understand the report](https://github.com/JasonShen2002/ssme-lite/blob/main/docs/guides/reports.md)
+- [API reference](https://github.com/JasonShen2002/ssme-lite/blob/main/docs/api.md)
+
+| Tutorial | What you will learn |
+| --- | --- |
+| [CivilComments](https://github.com/JasonShen2002/ssme-lite/blob/main/docs/tutorials/civilcomments.md) | Quickstart on the original paper's dataset and a qualitative reproduction check |
+| [PneumoniaMNIST](https://github.com/JasonShen2002/ssme-lite/blob/main/docs/tutorials/pneumoniamnist.md) | Main end-to-end demonstration: prediction, evaluation, ranking, and reporting |
+| [Transfer learning](https://github.com/JasonShen2002/ssme-lite/blob/main/docs/tutorials/transfer.md) | Frozen ResNet18 features, five classifier heads, and a shared evaluation interface |
+
+[View saved experiment reports](https://jasonshen2002.github.io/ssme-lite/).
+
+## Method and interpretation
+
+SSME-Lite follows Shanmugam et al., *Evaluating Multiple Models Using Labeled and Unlabeled Data*, NeurIPS 2025. It combines classifier probabilities through additive log-ratio features, weighted class-conditional kernel densities, and iterative posterior updates with observed labels clamped.
+
+Report intervals describe latent-label uncertainty conditional on the fitted posterior. Model contribution measures sensitivity to leaving out an input model. See the [method guide](https://github.com/JasonShen2002/ssme-lite/blob/main/docs/method.md) for definitions, implementation choices, and attribution. The current real-data demonstrations are binary classification tasks; the interface also accepts multiclass probabilities.
+
+## Development
+
+```bash
+python -m pip install -e '.[dev,docs]'
+python -m pytest -q
+mkdocs serve
+```
+
+See [Contributing](https://github.com/JasonShen2002/ssme-lite/blob/main/CONTRIBUTING.md) and the [release guide](https://github.com/JasonShen2002/ssme-lite/blob/main/docs/releasing.md).
+
+MIT licensed. Maintained by Junje Shen. Please cite the original SSME paper when using the method.

ssme_lite-0.1.0/README.md

@@ -0,0 +1,80 @@
+# SSME-Lite
+
+**Evaluate existing classifiers with a few labels and many unlabeled predictions.**
+
+SSME-Lite packages semi-supervised model evaluation into a small Python interface: provide model probabilities and partial labels, estimate performance, rank candidates, and export an interactive HTML report that opens offline.
+
+- **One estimator:** `fit()` followed by `report()`.
+- **Four metrics:** Accuracy, ROC AUC, average precision (`auprc`), and ECE.
+- **Flexible inputs:** precomputed probabilities or already fitted sklearn-style models.
+- **Portable outputs:** HTML, CSV, JSON, and metric plots.
+- **Reproducible experiments:** explicit seeds, shared splits, and optional model-influence diagnostics.
+- **Lightweight core:** no PyTorch dependency; transfer learning is optional.
+
+## Installation
+
+Python 3.10 or newer:
+
+```bash
+python -m pip install ssme-lite
+```
+
+To work from a checkout:
+
+```bash
+git clone https://github.com/JasonShen2002/ssme-lite.git
+cd ssme-lite
+python -m pip install -e '.[dev]'
+```
+
+## Minimal interface
+
+```python
+from ssme_lite import SSMEEstimator
+
+# scores: (samples, models), positive-class probabilities for binary tasks.
+# y_partial: one integer label per row; -1 means unknown.
+estimator = SSMEEstimator(random_state=0)
+estimator.fit(scores, y_partial, model_names=model_names)
+report = estimator.report(primary_metric="accuracy")
+print(report.ranking("accuracy"))
+report.save("results/my_report")
+```
+
+Every class must have at least one observed label. For multiclass tasks, use `(samples, models, classes)` probabilities and class indices `0..K-1`.
+
+For a complete example without external data, run `python examples/quickstart.py` from the checkout. It generates a small synthetic dataset, fits SSME, and saves `results/quickstart/report.html`. This example illustrates the interface, not an empirical performance claim.
+
+## Documentation and tutorials
+
+- [Documentation home](https://github.com/JasonShen2002/ssme-lite/blob/main/docs/index.md)
+- [Five-minute quickstart](https://github.com/JasonShen2002/ssme-lite/blob/main/docs/quickstart.md)
+- [Use your own models](https://github.com/JasonShen2002/ssme-lite/blob/main/docs/guides/inputs.md)
+- [Understand the report](https://github.com/JasonShen2002/ssme-lite/blob/main/docs/guides/reports.md)
+- [API reference](https://github.com/JasonShen2002/ssme-lite/blob/main/docs/api.md)
+
+| Tutorial | What you will learn |
+| --- | --- |
+| [CivilComments](https://github.com/JasonShen2002/ssme-lite/blob/main/docs/tutorials/civilcomments.md) | Quickstart on the original paper's dataset and a qualitative reproduction check |
+| [PneumoniaMNIST](https://github.com/JasonShen2002/ssme-lite/blob/main/docs/tutorials/pneumoniamnist.md) | Main end-to-end demonstration: prediction, evaluation, ranking, and reporting |
+| [Transfer learning](https://github.com/JasonShen2002/ssme-lite/blob/main/docs/tutorials/transfer.md) | Frozen ResNet18 features, five classifier heads, and a shared evaluation interface |
+
+[View saved experiment reports](https://jasonshen2002.github.io/ssme-lite/).
+
+## Method and interpretation
+
+SSME-Lite follows Shanmugam et al., *Evaluating Multiple Models Using Labeled and Unlabeled Data*, NeurIPS 2025. It combines classifier probabilities through additive log-ratio features, weighted class-conditional kernel densities, and iterative posterior updates with observed labels clamped.
+
+Report intervals describe latent-label uncertainty conditional on the fitted posterior. Model contribution measures sensitivity to leaving out an input model. See the [method guide](https://github.com/JasonShen2002/ssme-lite/blob/main/docs/method.md) for definitions, implementation choices, and attribution. The current real-data demonstrations are binary classification tasks; the interface also accepts multiclass probabilities.
+
+## Development
+
+```bash
+python -m pip install -e '.[dev,docs]'
+python -m pytest -q
+mkdocs serve
+```
+
+See [Contributing](https://github.com/JasonShen2002/ssme-lite/blob/main/CONTRIBUTING.md) and the [release guide](https://github.com/JasonShen2002/ssme-lite/blob/main/docs/releasing.md).
+
+MIT licensed. Maintained by Junje Shen. Please cite the original SSME paper when using the method.

ssme_lite-0.1.0/docs/api.md

@@ -0,0 +1,117 @@
+# API reference
+
+The stable public imports are:
+
+```python
+from ssme_lite import (
+    SSMEEstimator, PredictionMatrixGenerator,
+    SemiSupervisedSplit, EvaluationReport,
+)
+```
+
+## SSMEEstimator
+
+```python
+SSMEEstimator(density="kde", bandwidth="scott", max_iter=100,
+              tol=1e-3, labeled_weight=10.0, clip=1e-6,
+              random_state=0, n_jobs=1, batch_size=2048,
+              early_stopping=False)
+```
+
+| Parameter | Meaning |
+| --- | --- |
+| `density` | Currently only `"kde"` is supported |
+| `bandwidth` | `"scott"`, `"official"`, or a positive scalar in ALR space |
+| `max_iter` | Positive integer iteration count or early-stopping limit |
+| `tol` | Positive threshold for maximum posterior change |
+| `labeled_weight` | Positive weight on observed samples; unlabeled weight is 1 |
+| `clip` | Probability clipping threshold, strictly between 0 and 0.5 |
+| `random_state` | Seed for initialization and default report sampling |
+| `n_jobs` | joblib parallelism for class-density fitting; 1 is serial |
+| `batch_size` | Positive batch size for posterior evaluation |
+| `early_stopping` | Whether to stop when posterior change falls below `tol` |
+
+### fit
+
+```python
+fit(scores, y, scores_unlabeled=None, model_names=None,
+    initial_responsibilities=None)
+```
+
+Returns the fitted estimator. `scores` is binary `(N, M)` or full `(N, M, K)`. `y` has shape `(N,)`, contains class indices or `-1`, and includes every class among observed entries. If supplied, `scores_unlabeled` is appended after the first array. `model_names` contains unique names for the model axis; default names are `model_0`, etc.
+
+Advanced: `initial_responsibilities` is a nonnegative normalized `(N_total, K)` posterior used for controlled initialization. Observed labels are still clamped. Ordinary use should leave it unset.
+
+Invalid shapes, probabilities, labels, unsupported density choices, or duplicate names raise `ValueError`. The `official` bandwidth raises `ImportError` without statsmodels.
+
+### Fitted attributes
+
+| Attribute | Meaning |
+| --- | --- |
+| `posterior_` | `(N_total, K)` latent-class probabilities, with observed rows clamped |
+| `scores_`, `y_` | Normalized full probability tensor and aligned partial labels |
+| `model_names_`, `classes_` | Candidate names and class indices |
+| `labeled_indices_`, `unlabeled_indices_` | Indices in the combined fitted pool |
+| `bandwidth_`, `priors_` | Effective scalar bandwidth and fitted class priors |
+| `history_` | Per-iteration posterior-change, objective, and class summaries |
+| `n_iter_`, `converged_`, `stop_reason_` | Actual iterations, tolerance status, and stopping reason |
+
+### predict_proba
+
+```python
+predict_proba(scores)
+```
+
+Returns `(N_new, K)` posterior probabilities from the fitted densities. Input model and class axes must match the fit. This returns inferred labels for samples, not per-model performance. It does not clamp labels for new rows. Use `posterior_` when you need the fitted pool with observed-label clamping.
+
+### evaluate / report
+
+```python
+evaluate(metrics=("accuracy", "auc", "auprc", "ece"), n_draws=100,
+         confidence=0.95, target="all", random_state=None,
+         title=None, sample_ids=None, primary_metric=None,
+         contribution=False)
+report(**kwargs)
+```
+
+Returns `EvaluationReport`; `report` is an alias for `evaluate`. Metrics must be distinct supported keys. `n_draws >= 2`, `0 < confidence < 1`, and target is `"all"` or `"unlabeled"`. The default report seed inherits the estimator seed. `sample_ids`, if provided, identify all fitted rows. `primary_metric` must be among the requested metrics and defaults to the first. Contribution requires at least two models and some unlabeled samples.
+
+### contribution
+
+`contribution(n_draws=30)` returns a pandas DataFrame of leave-one-model-out diagnostics. `n_draws` is retained for compatibility; this diagnostic uses exact expected Accuracy and does not sample labels. See [interpretation](guides/reports.md).
+
+## PredictionMatrixGenerator
+
+```python
+PredictionMatrixGenerator(models, clip=1e-6, n_jobs=1)
+```
+
+`models` is a mapping of names to already fitted classifiers, or a sequence assigned automatic names. Each classifier must expose `classes_` and `predict_proba(X)`.
+
+- `fit(X=None, y=None)` inspects models and aligns class columns; returns self.
+- `transform(X, y_partial=None)` collects probabilities; optional labels record labeled/unlabeled indices.
+- `fit_transform(X, y=None)` combines the two steps without training classifiers.
+
+Attributes include `model_names_`, `classes_`, and, after transformation, `sample_ids_`. Binary output is `(N, M)` for `classes_[1]`; multiclass output is `(N, M, K)`.
+
+## SemiSupervisedSplit
+
+```python
+SemiSupervisedSplit(n_labeled, n_unlabeled, random_state=0)
+```
+
+Both counts must be positive integers. `indices(n_samples)` returns `labeled`, `unlabeled`, `estimation`, and `heldout` index arrays, requiring a nonempty held-out set. `partial_labels(y, parts)` returns the partial-label array aligned to `estimation`. `labeled_class_count(y, parts)` counts represented classes. Labels do not affect the permutation.
+
+## EvaluationReport
+
+Usually obtain this object from `estimator.report()`.
+
+- `table`: pandas DataFrame of estimates and uncertainty summaries.
+- `metadata`: settings and interpretation notes.
+- `data`: structured interactive-report payload.
+- `ranking(metric="accuracy")`: sorted DataFrame with a `rank` column; ECE ascending, other metrics descending.
+- `show()`: print estimates and fit status; return self.
+- `plot(metric="accuracy", path=None)`: return a Matplotlib figure and optionally save it.
+- `save(directory)`: create outputs and return the HTML `pathlib.Path`.
+
+See the [report guide](guides/reports.md) for table fields and saved-file definitions.

ssme_lite-0.1.0/docs/guides/cli.md

@@ -0,0 +1,41 @@
+# Command-line evaluation
+
+The CLI accepts a JSON configuration and can evaluate saved probabilities without model objects.
+
+## 1. Save inputs
+
+```python
+import numpy as np
+
+np.savez("predictions.npz", scores=scores, y=y_partial,
+         model_names=np.asarray(model_names, dtype=str))
+```
+
+Use the [input contract](inputs.md). Store names as a Unicode array, not a pickled object array. Unknown labels are `-1`.
+
+## 2. Write `evaluation.json`
+
+```json
+{
+  "source": "npz",
+  "mode": "evaluate",
+  "data": "predictions.npz",
+  "output": "results/evaluation",
+  "estimator": {"max_iter": 100, "random_state": 0},
+  "report": {"n_draws": 100, "primary_metric": "accuracy"}
+}
+```
+
+## 3. Run
+
+```bash
+ssme-lite evaluation.json
+```
+
+Paths resolve relative to the configuration file, not the shell's current directory. The command prints the output directory and saves the report plus `input_config.json`.
+
+For a downloadable example, run `python examples/quickstart.py` from the checkout: it also writes `results/quickstart/predictions.npz` and `evaluation.json`. Then run `ssme-lite results/quickstart/evaluation.json`.
+
+## Benchmark configurations
+
+The repository's `configs/` also includes synthetic and released-score benchmark configurations. Those run repeated experimental comparisons and have different data requirements and costs. The small NPZ example above is the recommended first CLI run. See `ssme-lite --help` for the command syntax.

ssme_lite-0.1.0/docs/guides/inputs.md

@@ -0,0 +1,67 @@
+# Use your own predictions
+
+## Input contract
+
+| Input | Shape / values | Meaning |
+| --- | --- | --- |
+| Binary scores | `(N, M)`, probabilities in `[0, 1]` | Probability of class index 1 |
+| Full probabilities | `(N, M, K)` | Class probabilities summing to one on the last axis |
+| Partial labels | `(N,)`, `-1` or integers `0..K-1` | Unknown labels or observed class indices |
+| Model names | `M` unique strings | Names shown in tables and reports |
+
+All models must predict the same samples in the same order and the same class set. Every class needs at least one observed label. Pass probabilities, not logits. Inputs are copied, clipped away from zero and one, and normalized internally.
+
+## Runnable sklearn example
+
+From the checkout root, run:
+
+```bash
+python examples/sklearn_models.py
+```
+
+This self-contained script generates data, trains logistic regression and a random forest on 300 training samples, and collects their predictions on 300 different samples. It then fits SSME with 20 observed labels and 180 unlabeled samples, reserving 100 held-out rows. The output is `results/sklearn_models/report.html`, including model-influence diagnostics. No external data or model weights are required.
+
+## Already fitted classifiers
+
+```python
+from ssme_lite import PredictionMatrixGenerator, SSMEEstimator
+
+# Each value is already trained, and exposes classes_ and predict_proba(X).
+generator = PredictionMatrixGenerator({"logistic": fitted_logistic, "forest": fitted_forest})
+scores = generator.fit_transform(X_evaluation)
+estimator = SSMEEstimator(random_state=0).fit(
+    scores, y_partial, model_names=generator.model_names_
+)
+report = estimator.report()
+```
+
+The generator aligns probability columns to the first model's `classes_`. Its `fit()` inspects models; it does not train them. Binary output uses `generator.classes_[1]` as the positive class.
+
+If original labels are strings or nonconsecutive numbers, encode observed labels as positions in `generator.classes_`. Reserve `-1` for unknown entries. For example, classes `['healthy', 'pneumonia']` map to `[0, 1]`.
+
+## Separate labeled and unlabeled arrays
+
+```python
+estimator.fit(
+    labeled_scores, labeled_y,
+    scores_unlabeled=unlabeled_scores,
+    model_names=model_names,
+)
+```
+
+This appends the unlabeled rows internally. Model and class dimensions must match. Reports default to all fitted rows; use `report(target="unlabeled")` to evaluate only the unlabeled portion.
+
+## Controlled benchmark splits
+
+When full labels are available for a benchmark, use `SemiSupervisedSplit` to hide them reproducibly:
+
+```python
+from ssme_lite import SemiSupervisedSplit
+
+splitter = SemiSupervisedSplit(20, 400, random_state=0)
+parts = splitter.indices(len(y))
+y_partial = splitter.partial_labels(y, parts)
+estimator.fit(scores[parts["estimation"]], y_partial, model_names=model_names)
+```
+
+The remaining samples form `parts['heldout']`. They are not used by the estimator. The splitter uses one uniform permutation and does not inspect labels to repair a missing-class draw. Check `splitter.labeled_class_count(y, parts)` when auditing benchmark splits, and document exclusions or additional annotation. Keep classifier training separate from this evaluation pool.

ssme_lite-0.1.0/docs/guides/reports.md

@@ -0,0 +1,49 @@
+# Understand the report
+
+```python
+report = estimator.report(n_draws=100, primary_metric="accuracy")
+report.ranking("accuracy")
+report.table
+report.save("results/evaluation")
+```
+
+## Metrics and rankings
+
+| Key | Quantity | Better direction |
+| --- | --- | --- |
+| `accuracy` | Fraction of correct class predictions | Higher |
+| `auc` | ROC area; macro one-vs-rest for multiclass | Higher |
+| `auprc` | Average precision; macro average for multiclass | Higher |
+| `ece` | Expected calibration error, 10 equal-width bins | Lower |
+
+`auprc` is computed with sklearn's average precision, not trapezoidal integration of a precision–recall curve. Binary ECE uses positive-class probabilities; multiclass ECE uses top-label confidence. Ranking uses the point estimate and assigns the same minimum rank to ties. Choose the metric appropriate to your task before comparing candidates.
+
+## Table columns
+
+`model`, `metric`, and `estimate` identify each estimate. `interval_low` and `interval_high` are conditional latent-label quantiles at the requested confidence level. `label_std` is the standard deviation across valid label draws; `mc_standard_error` quantifies numerical Monte Carlo error. `valid_draws` records how many finite metric values were available.
+
+Accuracy uses its exact conditional expectation, so its point estimate has zero Monte Carlo error. Other metric estimates average valid draws. Accuracy intervals are still obtained from sampled labels. AUC/AP draws missing a class are undefined and omitted; inspect `valid_draws` for small or imbalanced targets.
+
+Intervals condition on a fitted posterior. They do not include density-fit, bandwidth, or population sampling uncertainty. Increasing `n_draws` improves numerical precision; it does not account for those additional sources.
+
+## Files you can share
+
+| File | Purpose |
+| --- | --- |
+| `report.html` | Interactive report, with embedded assets and data; opens offline |
+| `metrics.csv` | Metric table for spreadsheets or further analysis |
+| `metadata.json` | Fit settings, target, seed, and interpretation notes |
+| `report_data.json` | Structured data powering the report |
+| Metric PNGs | Static plots for presentations |
+| `contribution.csv` | Produced when model contribution is enabled |
+
+The report summarizes prediction and posterior data. Inspect its contents before sharing outputs from sensitive datasets.
+
+## Model influence
+
+```python
+report = estimator.report(contribution=True)
+report.save("results/with_contribution")
+```
+
+This refits once per removed model, holding visible labels, scalar bandwidth, iteration count, and full-fit initialization fixed. Total variation measures the change in the unlabeled posterior; the report also shows changes to the remaining models' Accuracy estimates. It describes conditional influence, not a causal effect or a guaranteed benefit from dropping a model. Runtime increases by roughly one fit per candidate.

ssme_lite-0.1.0/docs/index.md

@@ -0,0 +1,32 @@
+# SSME-Lite
+
+## From classifier probabilities to an evaluation report
+
+SSME-Lite helps you evaluate several already trained classifiers when labels are scarce. It combines a small labeled sample with the models' predictions on unlabeled samples, then produces metric estimates, rankings, and an offline report.
+
+**The workflow:** collect probabilities → reveal a few labels → fit `SSMEEstimator` → inspect and export results.
+
+## Start here
+
+1. [Install the package](installation.md).
+2. [Run the five-minute quickstart](quickstart.md), with no downloads or model weights.
+3. [Connect your own classifiers](guides/inputs.md).
+4. [Read the report](guides/reports.md) and choose your target metric.
+
+## Learn through real experiments
+
+| Tutorial | Role | Inputs |
+| --- | --- | --- |
+| [CivilComments](tutorials/civilcomments.md) | Original-paper quickstart and reproduction check | Released classifier scores |
+| [PneumoniaMNIST](tutorials/pneumoniamnist.md) | Main end-to-end demonstration | Images and seven released models, or saved probabilities |
+| [Transfer learning](tutorials/transfer.md) | Adapt the workflow to newly trained heads | Frozen ImageNet backbone and five sklearn heads |
+
+The core package consumes probabilities. Training models and downloading datasets belong to the tutorials, so you can use the estimator without a deep-learning framework.
+
+## Explore further
+
+- [Command-line evaluation](guides/cli.md): run from an NPZ file and JSON configuration.
+- [Method and interpretation](method.md): algorithm, metrics, intervals, and model influence.
+- [API reference](api.md): public classes, defaults, inputs, and outputs.
+- [Troubleshooting](troubleshooting.md): common setup and data-alignment problems.
+- [Saved interactive reports](https://jasonshen2002.github.io/ssme-lite/): inspect results before running experiments.

ssme_lite-0.1.0/docs/installation.md

@@ -0,0 +1,52 @@
+# Installation
+
+## Core package
+
+Use Python 3.10 or newer, preferably in a dedicated virtual environment:
+
+```bash
+python -m venv .venv
+# macOS / Linux
+source .venv/bin/activate
+# Windows PowerShell: .venv\Scripts\Activate.ps1
+python -m pip install --upgrade pip
+python -m pip install ssme-lite
+python -c "from ssme_lite import SSMEEstimator; print(SSMEEstimator())"
+```
+
+The core uses NumPy, SciPy, pandas, scikit-learn, Matplotlib, and joblib. It does not require PyTorch, model weights, or network access at evaluation time.
+
+## Optional features
+
+```bash
+# Reference-code bandwidth rule
+python -m pip install 'ssme-lite[official]'
+# Frozen-backbone transfer-learning dependencies
+python -m pip install 'ssme-lite[transfer]'
+```
+
+`official` adds statsmodels; it does not download the original research repository. `transfer` adds PyTorch, torchvision, and Pillow; experiment scripts remain in the source repository.
+
+## Tutorials and development
+
+```bash
+git clone https://github.com/JasonShen2002/ssme-lite.git
+cd ssme-lite
+python -m pip install -e '.[dev,docs]'
+python examples/quickstart.py
+mkdocs serve
+```
+
+Run repository commands from the checkout root unless a tutorial says otherwise. Notebooks additionally require Jupyter: `python -m pip install jupyterlab`. The full PneumoniaMNIST inference path also requires `ai-edge-litert`; availability depends on your Python version and platform. Its saved-probability path uses only the core package.
+
+## Conda environments
+
+You can install the PyPI distribution inside an activated conda environment:
+
+```bash
+conda create -n ssme python=3.11 pip
+conda activate ssme
+python -m pip install ssme-lite
+```
+
+A draft recipe is included in `conda/`; this does not mean the package has been accepted into conda-forge.

ssme_lite-0.1.0/docs/method.md

@@ -0,0 +1,35 @@
+# Method and interpretation
+
+## Algorithm
+
+SSME-Lite implements a semi-supervised evaluation workflow based on Shanmugam et al. (2025). Candidate classifiers are already trained. Their probability vectors form the inputs to the evaluator.
+
+1. Clip and normalize probabilities, then apply an additive log-ratio transformation with the final class as reference.
+2. Initialize latent classes by sampling from the mean candidate probabilities; clamp observed labels.
+3. Fit weighted class-conditional Gaussian kernel densities and update class priors.
+4. Update class posteriors and clamp observed labels again.
+5. Repeat for the requested number of iterations, then evaluate candidates under the fitted posterior.
+
+Observed samples have default weight 10; unlabeled samples have weight 1. The default is 100 fixed iterations (`early_stopping=False`). With early stopping enabled, the largest posterior change is compared with `tol=1e-3`. `history_`, `n_iter_`, `converged_`, and `stop_reason_` expose fit diagnostics. Fixed-bandwidth KDE updates are an EM-style procedure; the recorded objective is a diagnostic rather than a promised monotonic optimization trace.
+
+## Bandwidth choices
+
+- `"scott"` (default): mean feature standard deviation multiplied by the multivariate Scott sample-size factor, with a minimum scalar bandwidth of `1e-3`.
+- `"official"`: the smallest per-feature statsmodels normal-reference bandwidth divided by four, following the public reference implementation; install the `official` extra.
+- A positive number: use that fixed scalar bandwidth in ALR space.
+
+`official` identifies a reference-code bandwidth rule, not bitwise equivalence of every experiment. For example, the package uses equal-width ECE bins, whereas the paper reports equal-frequency bins.
+
+## What is evaluated?
+
+By default, reports estimate performance over the fitted estimation pool, including observed and unobserved rows. `target="unlabeled"` restricts the target to unlabeled rows. Held-out benchmark metrics are computed separately and must be labeled as such. The real-data tutorials demonstrate binary tasks; multiclass input support alone is not evidence of benchmark performance on multiclass datasets.
+
+The model-influence diagnostic is a controlled leave-one-model-out sensitivity analysis. See [report interpretation](guides/reports.md) for interval and influence definitions.
+
+## Attribution
+
+SSME-Lite is an independent packaging and workflow implementation, not the authors' official distribution.
+
+Divya Shanmugam, Shuvom Sadhuka, Manish Raghavan, John Guttag, Bonnie Berger, and Emma Pierson. **Evaluating Multiple Models Using Labeled and Unlabeled Data.** Advances in Neural Information Processing Systems 38, 2025, pp. 30612–30648. [Paper](https://proceedings.neurips.cc/paper_files/paper/2025/hash/2c0e78cc177dfeca260ef990a8c99209-Abstract-Conference.html) · [Official code](https://github.com/divyashan/SSME).
+
+The software is MIT licensed. Dataset and checkpoint terms remain those of their original providers; these assets are not bundled in the Python distribution.