cli-modelarium 0.1.3__tar.gz

cli_modelarium-0.1.3/.gitattributes

@@ -0,0 +1,39 @@
+# Auto detect text files and perform LF normalization
+* text=auto eol=lf
+
+# Explicitly declare text files
+*.py text eol=lf
+*.pyi text eol=lf
+*.md text eol=lf
+*.toml text eol=lf
+*.cfg text eol=lf
+*.ini text eol=lf
+*.yml text eol=lf
+*.yaml text eol=lf
+*.json text eol=lf
+*.txt text eol=lf
+*.csv text eol=lf
+*.sh text eol=lf
+*.bash text eol=lf
+*.gitignore text eol=lf
+*.gitattributes text eol=lf
+
+# Documentation
+LICENSE text eol=lf
+NOTICE text eol=lf
+README* text eol=lf
+CHANGELOG* text eol=lf
+CONTRIBUTING* text eol=lf
+SECURITY* text eol=lf
+
+# Declare binary files (no line ending conversion)
+*.png binary
+*.jpg binary
+*.jpeg binary
+*.gif binary
+*.ico binary
+*.svg binary
+*.pdf binary
+*.zip binary
+*.tar.gz binary
+*.whl binary

cli_modelarium-0.1.3/.github/FUNDING.yml

@@ -0,0 +1 @@
+github: [lavellehatcherjr]

cli_modelarium-0.1.3/.github/dependabot.yml

@@ -0,0 +1,13 @@
+version: 2
+updates:
+  - package-ecosystem: "pip"
+    directory: "/"
+    schedule:
+      interval: "weekly"
+    open-pull-requests-limit: 5
+
+  - package-ecosystem: "github-actions"
+    directory: "/"
+    schedule:
+      interval: "weekly"
+    open-pull-requests-limit: 5

cli_modelarium-0.1.3/.github/workflows/ci.yml

@@ -0,0 +1,60 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+  workflow_dispatch:
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  test:
+    name: Test (${{ matrix.os }}, Python ${{ matrix.python-version }})
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest, windows-latest]
+        python-version: ["3.11", "3.12", "3.13"]
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v6
+
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v6
+        with:
+          python-version: ${{ matrix.python-version }}
+          cache: 'pip'
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e ".[dev,schema]"
+
+      - name: Run tests
+        run: python -m pytest tests/ -v
+
+  lint:
+    name: Lint (ruff)
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v6
+
+      - name: Set up Python
+        uses: actions/setup-python@v6
+        with:
+          python-version: "3.13"
+          cache: 'pip'
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e ".[dev,schema]"
+
+      - name: Run ruff
+        run: ruff check .

cli_modelarium-0.1.3/.gitignore

@@ -0,0 +1,225 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[codz]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#   Usually these files are written by a python script from a template
+#   before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py.cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+# Pipfile.lock
+
+# UV
+#   Similar to Pipfile.lock, it is generally recommended to include uv.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+# uv.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+# poetry.lock
+# poetry.toml
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#   pdm recommends including project-wide configuration in pdm.toml, but excluding .pdm-python.
+#   https://pdm-project.org/en/latest/usage/project/#working-with-version-control
+# pdm.lock
+# pdm.toml
+.pdm-python
+.pdm-build/
+
+# pixi
+#   Similar to Pipfile.lock, it is generally recommended to include pixi.lock in version control.
+# pixi.lock
+#   Pixi creates a virtual environment in the .pixi directory, just like venv module creates one
+#   in the .venv directory. It is recommended not to include this directory in version control.
+.pixi
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# Redis
+*.rdb
+*.aof
+*.pid
+
+# RabbitMQ
+mnesia/
+rabbitmq/
+rabbitmq-data/
+
+# ActiveMQ
+activemq-data/
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.envrc
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#   JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#   be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#   and can be added to the global gitignore or merged into this file.  For a more nuclear
+#   option (not recommended) you can uncomment the following to ignore the entire idea folder.
+.idea/
+
+# Abstra
+#   Abstra is an AI-powered process automation framework.
+#   Ignore directories containing user credentials, local state, and settings.
+#   Learn more at https://abstra.io/docs
+.abstra/
+
+# Visual Studio Code
+#   Visual Studio Code specific template is maintained in a separate VisualStudioCode.gitignore 
+#   that can be found at https://github.com/github/gitignore/blob/main/Global/VisualStudioCode.gitignore
+#   and can be added to the global gitignore or merged into this file. However, if you prefer, 
+#   you could uncomment the following to ignore the entire vscode folder
+.vscode/
+# Temporary file for partial code execution
+tempCodeRunnerFile.py
+
+# Ruff stuff:
+.ruff_cache/
+
+# PyPI configuration file
+.pypirc
+
+# Marimo
+marimo/_static/
+marimo/_lsp/
+__marimo__/
+
+# Streamlit
+.streamlit/secrets.toml
+
+# OS / editor junk
+.DS_Store
+.DS_Store?
+._*
+Thumbs.db
+Desktop.ini

cli_modelarium-0.1.3/CHANGELOG.md

@@ -0,0 +1,149 @@
+# Changelog
+
+All notable changes to Cli Modelarium will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.1.3] - 2026-05-28
+
+### Added
+
+- Bootstrap confidence intervals on per-cell means via `scipy.stats.bootstrap`. Auto-enabled when `--runs > 1`.
+- New CLI flags on `compare`:
+  - `--confidence-intervals` / `--no-confidence-intervals` (auto-enabled with `--runs > 1`)
+  - `--ci-level FLOAT` (default `0.95`)
+  - `--ci-method {bca,percentile,basic}` (default `bca` - publication-grade)
+  - `--bootstrap-resamples INT` (default `5000`, min `100`)
+  - `--bootstrap-seed INT` (required for reproducible CIs)
+- New test choices on `--significance-test`:
+  - `paired-t` - paired t-test via `scipy.stats.ttest_rel` (more statistical power for same-prompt comparisons)
+  - `wilcoxon-signed` - Wilcoxon signed-rank via `scipy.stats.wilcoxon` (non-parametric paired)
+- McNemar's test for paired binary outcomes. Auto-triggers when `--check-hallucination` is set with `--runs > 1` and 2+ models. Uses exact binomial test (`scipy.stats.binomtest`) for small discordant counts or Edwards continuity-corrected chi-square (`scipy.stats.chi2.sf`) for larger samples - NOT `scipy.stats.chi2_contingency` on the full 2×2 table (which would compute a test of independence, not McNemar).
+- Bootstrap CIs on Cohen's d effect sizes via paired/independent bootstrap.
+- `mcnemar_tests` array in JSON output when applicable.
+- `methodology` block in JSON output recording bootstrap parameters, scipy version, and seed for reproducibility.
+- Additive CI columns in CSV output (`latency_ms_ci_low`, `latency_ms_ci_high`, etc.) when CIs are enabled.
+- Additive Markdown sections: "Bootstrap confidence intervals", "Statistical significance tests", "Binary outcome significance (McNemar)", "Statistical methodology".
+- New public functions in `cli_modelarium.run_statistics`:
+  - `ConfidenceInterval`, `McNemarResult` dataclasses
+  - `bootstrap_ci()` - thin scipy wrapper with degenerate-data handling
+  - `paired_t_test()`, `wilcoxon_signed_rank()`
+  - `mcnemar_test()` - Edwards-corrected or exact binomial McNemar
+  - `compute_significance_with_ci()` - like `compute_pairwise_significance` plus CIs on Cohen's d
+  - `compute_stats_with_cis()` - CIs on per-model metric means
+  - `compute_mcnemar_pairwise()` - pairwise McNemar over hallucination pass/fail
+- New private helpers ensuring paired tests align by `run_index` even when failures are asymmetric:
+  - `_extract_paired_metric_samples()`
+  - `_align_paired_samples()`
+
+### Changed
+
+- `SignificanceResult` dataclass extended with seven optional fields (all default `None`): `bootstrap_ci_low`, `bootstrap_ci_high`, `bootstrap_method`, `bootstrap_resamples`, `bootstrap_seed`, `effect_size_ci_low`, `effect_size_ci_high`. v0.1.2-style positional instantiation continues to work unchanged.
+- Output formatters extended so CSV, Markdown, and JSON all receive significance, CI, McNemar, and methodology data (previously only JSON received significance results - a v0.1.2 wiring gap).
+- `_emit_batch_results` threads the new parameters into every formatter branch.
+
+### Dependencies
+
+- No new runtime dependencies (uses scipy 1.17 already in v0.1.2).
+- `NOTICE` unchanged - scipy is already attributed.
+- Python version unchanged (still `>=3.11` from v0.1.2).
+
+## [0.1.2] - 2026-05-28
+
+### ⚠️ Breaking Changes
+
+- **Minimum Python version is now 3.11** (was 3.10).
+  - Reason: scipy 1.17+ is a new runtime dependency for statistical significance testing, and scipy 1.17 requires Python 3.11+.
+  - Python 3.10 users can continue using cli-modelarium v0.1.1, which remains available on PyPI.
+  - Python 3.10 reaches end-of-life in October 2026.
+
+### Added
+
+- Pairwise statistical significance testing on the `compare` command. Auto-enabled when `--runs > 1` with 2+ models.
+- New CLI flags on `compare`:
+  - `--significance` / `--no-significance` (auto-enabled with `--runs > 1` and 2+ models)
+  - `--significance-threshold FLOAT` (default: `0.05`)
+  - `--significance-test {welch,mann-whitney}` (default: `welch`)
+  - `--correction {none,bonferroni,holm}` (default: `bonferroni`)
+  - `--significance-metric {score,latency_ms,output_tokens,cost_usd}` (default: `score` when judging, else `latency_ms`)
+- Welch's t-test via `scipy.stats.ttest_ind(equal_var=False)`.
+- Mann-Whitney U test via `scipy.stats.mannwhitneyu` with continuity correction.
+- Cohen's d effect size with conventional interpretation bands (`negligible` / `small` / `medium` / `large`), implemented in pure stdlib.
+- Bonferroni and Holm-Bonferroni multiple-comparison corrections, implemented in pure stdlib with monotone enforcement.
+- JSON output now includes a `significance_tests` array when significance testing was performed. When significance is disabled or trivially absent, the JSON schema is unchanged (additive only).
+- Display strategy: single-line summary for 2 models, matrix table for 3-5 models, top-K significant pairs for 6+ models (full matrix available in JSON).
+- New functions in `cli_modelarium.run_statistics`:
+  - `SignificanceResult` dataclass
+  - `compute_pairwise_significance()`
+  - `welch_t_test()`, `mann_whitney_u_test()`
+  - `cohens_d()`, `cohens_d_interpretation()`
+  - `bonferroni_correct()`, `holm_correct()`
+
+### Changed
+
+- `pyproject.toml`: `requires-python` bumped to `>=3.11`.
+- `pyproject.toml`: classifiers updated - removed Python 3.10, added 3.13 and 3.14.
+- `pyproject.toml`: `[tool.ruff].target-version` bumped to `py311`.
+- `NOTICE`: added attributions for scipy, numpy, and the bundled native libraries (OpenBLAS, LAPACK, libquadmath).
+- README: new "System Requirements" section and statistical-significance documentation.
+
+### Fixed
+
+- Resolved a latent inconsistency: `tests/test_jsonschema_optional.py` already imported `tomllib` (Python 3.11+ stdlib only) while the project declared 3.10 support. The Python bump retroactively fixes this.
+
+### Dependencies
+
+- Added: `scipy>=1.17,<2.0` (pulls `numpy>=1.26.4` as a transitive dependency).
+
+## [0.1.1] - 2026-05-27
+
+### Added
+
+- `--runs N` flag on the `compare` command for statistical reproducibility analysis. Runs each (model, temperature, system_prompt) combination N times (1-100) and displays mean/median/stdev/CV of timing and tokens, cost totals, output frequency analysis, mode output, and output diversity.
+- `--show-all-runs` flag to override the auto-collapse heuristic when many concurrent display panels would be created.
+- New module `src/cli_modelarium/run_statistics.py` with `RunStats` dataclass and `compute_run_stats()` function for pure-stdlib statistical analysis.
+- Hallucination rate calculation when `--check-hallucination` is combined with `--runs N`. Reports "N of M runs flagged as High risk" with the aggregate hallucination rate.
+- Cost warning when `--runs N` is used without `--max-cost` (helps prevent unexpected spend).
+
+### Changed
+
+- `compare` command's display path branches when `runs > 1` to show statistical summary instead of per-run details. When `runs == 1` (default), behavior is byte-identical to v0.1.0.
+- `run_streaming_comparison()` accepts new keyword parameters `runs: int = 1` and `show_all_runs: bool = False`. Default values preserve existing behavior.
+- `StreamState` dataclass has new field `run_index: int = 0`. Default value preserves all existing test expectations.
+- `BatchResult` dataclass has new field `run_index: int = 0`. Only emitted in CSV/JSON/Markdown output when the surrounding `runs` parameter > 1.
+- Live streaming display auto-collapses when total concurrent tasks exceed 12 (configurable via `--show-all-runs`).
+- LLM-as-judge with `--runs N`:
+  - Default (with `--judge` or `--judges`): mode-only judging (one judge call per cell, expanded to every run in the cell)
+  - With `--check-hallucination`: per-run judging (computes hallucination rate)
+- JSON output schema additive when `runs > 1`: new `total_runs` and `stats_by_cell` top-level fields, plus `run_index` per result. When `runs == 1`, schema is byte-identical to v0.1.0.
+- CSV output adds `run_index` column when `runs > 1`. When `runs == 1`, columns are unchanged.
+- Markdown output adds a "Per-cell statistical summary" section when `runs > 1`. When `runs == 1`, output is unchanged.
+
+### Fixed
+
+- N/A (no bug fixes in this release; only additions)
+
+## [0.1.0] - 2026-05-25
+
+### Added
+
+- Initial v0.1.0 release
+- 8 cloud provider integrations: OpenAI, Anthropic, Google, xAI, DeepSeek, Mistral, Groq, OpenRouter
+- Local model support: Ollama, LM Studio, vLLM, llama.cpp via OpenAI-compatible API
+- Parallel streaming with TTFT (Time To First Token) tracking
+- Multi-prompt batch mode with CSV, JSON, and Markdown output formats
+- System prompt support: single, multiple (comparison), and file-based
+- LLM-as-a-judge scoring with panel mode and self-evaluation skip
+- Deterministic assertions with 10 types (`contains`, `not_contains`, `regex`, `equals`, `json_valid`, `json_schema`, `min_length_chars`, `max_length_chars`, `latency_under`, `cost_under`)
+- CI/CD exit codes: 0 = success, 1 = assertion failure, 2 = call failure (call failures dominate)
+- Hallucination detection preset with optional reference facts and worst-wins panel aggregation
+- OS-native keychain integration via `keyring`
+- API key format validation for 8 providers
+- Error message redaction prevents key leakage
+- Localhost-only validation for local model URLs
+- Cross-platform support: macOS, Windows 10+/ARM, Linux
+- Rate limit handling: 429 retry with exponential backoff, 529 (Anthropic overloaded) with longer backoff
+- `retry-after` header honored when present
+- Per-provider semaphores for concurrent request management
+- Atomic file writes for output integrity
+- Apache 2.0 License with proper NOTICE attribution

cli_modelarium-0.1.3/CONTRIBUTING.md

@@ -0,0 +1,76 @@
+# Contributing to Cli Modelarium
+
+Thanks for your interest in Cli Modelarium - a terminal tool for
+statistically rigorous LLM comparison across multiple providers.
+
+## Quick Links
+
+- GitHub: https://github.com/lavellehatcherjr/cli-modelarium
+- Issues: https://github.com/lavellehatcherjr/cli-modelarium/issues/new
+- Maintainer: Lavelle Hatcher Jr ([@lavellehatcherjr](https://github.com/lavellehatcherjr)) - Creator & Maintainer
+
+## How to Contribute
+
+Cli Modelarium is maintained by one person. I personally review and decide on
+every contribution. Not everything will be merged - so for anything beyond a
+small fix, please open an issue first so we can agree on the approach before
+you spend time writing code. This saves your effort as much as mine.
+
+1. **Typos and small bug fixes** -> open a pull request directly.
+2. **New features or larger changes** -> open an issue first to discuss the
+   idea. Please don't send a large PR without checking in - it may not fit the
+   direction of the project, and I'd rather save you the work.
+3. **Refactor-only or style-only PRs** -> please don't open these unless I've
+   asked for them as part of a specific fix.
+4. **Questions** -> open an issue and I'll respond when I can.
+
+Because I'm a solo maintainer, reviews can take some time. Thank you for your
+patience, and thank you for helping make the project better.
+
+## Development Setup
+
+Clone the repository and install it with the development extras:
+
+```bash
+git clone https://github.com/lavellehatcherjr/cli-modelarium.git
+cd cli-modelarium
+python -m venv .venv
+source .venv/bin/activate
+pip install -e ".[dev,schema]"
+```
+
+This requires Python 3.11 or newer.
+
+## Running Tests and Lint
+
+Before opening a pull request, please run the test suite and the linter so
+your change is easy to review:
+
+```bash
+# Run the full test suite
+python -m pytest tests/
+
+# Run the linter on your changes
+ruff check .
+```
+
+All tests should pass. Please make sure your change does not add new lint
+warnings. (A few pre-existing lint items are being cleaned up separately, so
+don't worry about those.)
+
+## What Makes a Good PR
+
+- Keep it focused - one logical change per PR.
+- Include or update tests for any behavior you change.
+- Match the existing code style.
+- Describe what the change does and why in the PR description.
+
+## Security Issues
+
+See [SECURITY.md](SECURITY.md) for reporting security vulnerabilities. Do
+NOT open public issues for security problems.
+
+## License
+
+By contributing, you agree that your contributions will be licensed under the
+project's Apache 2.0 license.