evalring 0.2.0__tar.gz

evalring-0.2.0/.env.example

@@ -0,0 +1,28 @@
+# EvalRing configuration. Copy to .env and fill in; .env is gitignored.
+# Full reference: docs/CONFIGURATION.md
+
+# --- Provider credentials --------------------------------------------------
+# Vendor-neutral form. Works with any OpenAI-compatible chat-completions
+# endpoint and takes precedence over the provider-specific variables below.
+EVALRING_API_KEY=
+# Leave EVALRING_BASE_URL unset for OpenAI itself. Set it for OpenRouter, a
+# self-hosted gateway, vLLM, Ollama, or any other compatible endpoint.
+EVALRING_BASE_URL=
+# Default model when a script or agent does not name one explicitly.
+EVALRING_MODEL=
+
+# --- Provider-specific alternatives ---------------------------------------
+# Any one of these is enough on its own. Checked in this order after
+# EVALRING_API_KEY: OPENAI_API_KEY, OPENROUTER_API_KEY, OPEN_ROUTER_KEY,
+# RADIUM_API_KEY.
+# OPENAI_API_KEY=
+# OPENROUTER_API_KEY=
+
+# --- Behaviour -------------------------------------------------------------
+# Transport used by OpenAIAgent: "litellm" (default) or "openai".
+# EVALRING_LLM_TRANSPORT=litellm
+# Per-request timeout in seconds. 0 or negative disables the timeout.
+# OPENAI_REQUEST_TIMEOUT_S=120
+# Directory that holds the _EvalRing/ cache and run artifacts.
+# Defaults to the current working directory.
+# EVALRING_WORKSPACE=

evalring-0.2.0/AGENTS.md

@@ -0,0 +1,79 @@
+# Working in this repository
+
+Orientation for coding agents and for anyone reading the codebase for the first
+time. The documents under [`docs/`](docs/) are the authoritative description of
+this system and ship with the code — read them before the source, and update
+them in the same change that alters behaviour.
+
+## Where to read first
+
+| Question | Document |
+| --- | --- |
+| What is this and how is it built? | [`docs/DESIGN_SPEC.md`](docs/DESIGN_SPEC.md) |
+| What can I call from Python? | [`docs/API_REFERENCE.md`](docs/API_REFERENCE.md) |
+| How do I point it at a model? | [`docs/CONFIGURATION.md`](docs/CONFIGURATION.md) |
+| What does the CLI do? | [`docs/CLI.md`](docs/CLI.md) |
+| How do I run a real evaluation? | [`docs/USAGE.md`](docs/USAGE.md) |
+| Which datasets exist and what may I do with them? | [`docs/DATA.md`](docs/DATA.md) |
+| How do I contribute a change? | [`CONTRIBUTING.md`](CONTRIBUTING.md) |
+| How do I cut a release? | [`docs/RELEASING.md`](docs/RELEASING.md) |
+
+If a document disagrees with the code, the code is what runs — fix the document
+and say so in your summary.
+
+## Layout
+
+```
+src/EvalRing/        the installable package; the only thing that ships
+  config.py          credential and model resolution for every component
+  logging_utils.py   logger namespace and entry-point logging setup
+  cli.py             the `evalring` console script
+  dataset/           BaseDataset, DataSample, JSON/CSV/DataFrame readers
+  agent/             BaseAgent, AgentResponse, Mock/RuleBased/OpenAI agents,
+                     classification parsing, error classification
+  evaluator/         BaseEvaluator, ClassificationEvaluator, llm_judge/
+  utils/             suite runner, response cache, model lists, charts
+tests/               offline test suite
+examples/            runnable evaluations; NOT packaged, NOT importable
+docs/                authoritative documentation
+```
+
+## Ground rules
+
+These are enforced by CI, so a change that breaks one will fail before review.
+
+1. **`import EvalRing` must work with core dependencies only.** `openai`,
+   `litellm`, `matplotlib`, and `nest_asyncio` are optional. Import them inside
+   the function that needs them and raise an `ImportError` naming the extra.
+2. **No `print()` inside `src/EvalRing/`**, except machine-readable output the
+   user explicitly asked the CLI for. Use `get_logger(__name__)`.
+3. **No provider hard-coded anywhere.** Credentials and base URLs come from
+   `EvalRing.config.resolve_credentials()`. Never read a key from `os.environ`
+   directly, and never put an endpoint in a default argument.
+4. **Tests stay offline.** No network in the default suite. Use a scripted
+   `BaseAgent` subclass, not SDK mocks.
+5. **Nothing but `EvalRing` goes in the wheel.** `examples/` and `tests/` are
+   excluded; CI asserts the wheel's top-level contents.
+6. **Do not commit datasets or credentials.** See
+   [`docs/DATA.md`](docs/DATA.md) and [`SECURITY.md`](SECURITY.md).
+
+## Local commands
+
+```bash
+pip install -e ".[dev]"
+
+ruff check src tests
+ruff format src tests
+mypy
+pytest
+```
+
+Use `.agent_temp/` for scratch files and `.log/` for log output. Both are
+gitignored. Do not leave working files in the repository root.
+
+## Sensitive material
+
+`examples/suicide_detection/` and `examples/sample/med_note/` involve
+suicide-risk classification and clinical notes. Read
+[`docs/DATA.md`](docs/DATA.md) before touching them: the datasets carry
+redistribution conditions, and the outputs are not clinical tools.

evalring-0.2.0/CHANGELOG.md

@@ -0,0 +1,99 @@
+# Changelog
+
+All notable changes to this project are documented here. The format follows
+[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project
+follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.2.0] - 2026-09-01
+
+First release prepared for public use. The framework's behaviour is largely
+unchanged; this release makes the package honest about what it does, removes
+vendor-specific defaults, and adds the tests and CI a published library needs.
+
+### Added
+
+- `evalring` command-line interface with four subcommands: `info` (dataset
+  statistics), `check` (report the resolved provider configuration without
+  printing secrets), `models` (generate a model list), and `run-suite`
+  (multi-model evaluation). Previously the entry point was declared but the
+  module did not exist.
+- `EvalRing.config` with `resolve_credentials()` and `resolve_model_name()`, a
+  single documented precedence order for API keys, endpoints, and model names
+  across every component.
+- `EvalRing.logging_utils` with `get_logger()` and `configure_logging()`.
+- `py.typed` marker, so downstream projects get type information.
+- Test suite covering credentials, datasets, agents, evaluation, metrics,
+  caching, and the CLI. Every test runs offline.
+- CI across Python 3.10-3.13 on Linux, plus Windows and macOS: lint, format,
+  types, tests, a core-only install check, and a wheel-contents assertion.
+- Release workflow publishing to PyPI via Trusted Publishing on a version tag.
+- `GlobalCache.reset_instance()` and the `EVALRING_WORKSPACE` variable, so the
+  cache location can be controlled instead of always following the working
+  directory.
+- Documentation: `CONFIGURATION.md`, `API_REFERENCE.md`, `CLI.md`, `DATA.md`,
+  `RELEASING.md`, plus `AGENTS.md`, `SECURITY.md`, `CODE_OF_CONDUCT.md`, and
+  `.env.example`.
+
+### Changed
+
+- **Provider resolution is vendor-neutral.** `EVALRING_API_KEY` and
+  `EVALRING_BASE_URL` are the recommended variables. `OPENAI_API_KEY`,
+  `OPENROUTER_API_KEY`, `OPEN_ROUTER_KEY`, and `RADIUM_API_KEY` still work, in
+  that order. No private endpoint is used as a default any more; a provider's
+  base URL applies only when that provider's key is the one selected.
+- **The library logs instead of printing.** Modules under `EvalRing/` emit
+  records on the `EvalRing` logger hierarchy and no longer write to stdout or
+  configure the root logger. Entry points call `configure_logging()`.
+- Examples moved from `src/example/` to `examples/` and are no longer packaged.
+  Installing EvalRing previously also installed a top-level `example` package.
+- Dependencies now reflect what the code imports. Core: `pandas`, `tqdm`,
+  `python-dotenv`, `requests`. Optional extras: `llm` (litellm, openai, httpx),
+  `viz` (matplotlib), `datagen` (openai, nest-asyncio), `all`, `dev`.
+- Minimum Python is 3.10. The previously declared 3.8 floor was never valid:
+  `dataset/base.py` uses PEP 604 annotations evaluated at runtime.
+- `BaseDataset.load_data()` declares `text_field`, `label_field`, and
+  `id_field` so concrete readers no longer widen the base signature.
+- `run_suite()` takes `base_class=None` by default and forwards
+  `--base-class` only when set, instead of defaulting to a task-specific label.
+- `run_suite()` skips its interactive confirmation when stdin is not a TTY,
+  so automated runs cannot hang.
+- Package metadata: distribution renamed to `evalring` (the import name stays
+  `EvalRing`), real repository URLs, single version source in
+  `EvalRing.__version__`.
+
+### Removed
+
+- `setup.py` and `requirements.txt`. `pyproject.toml` is the single source of
+  build configuration and dependencies.
+- `EvalRing.examples`, a module that could not be imported: it referenced
+  `EvalRing.base` and `EvalRing.implementations`, neither of which exists.
+- The `sys.path` manipulation that `EvalRing.utils.suite_runner` performed at
+  import time, and the equivalent bootstrapping in the example scripts. Install
+  the package instead: `pip install -e .`.
+- Unused declared dependencies: `numpy`, `scikit-learn`, `seaborn`, `pyyaml`.
+  The wrong `dotenv` distribution was replaced with `python-dotenv`.
+
+### Fixed
+
+- The README documented an API that did not exist (`Evaluator`,
+  `ClassificationMetrics`, `load_dataset`, `create_agent`, `EvalRing.agents`,
+  `EvalRing.datasets`, and a CLI). It now documents the real API, verified by a
+  test.
+- A hardcoded absolute developer path in the suicide-detection suite runner.
+- A duplicated `--ignore-errors` flag in the suite runner's subprocess command.
+- `zip()` over ground truth and predictions is now strict, so a length mismatch
+  raises instead of silently truncating the metric computation.
+- Trailing-whitespace, unused-variable, and un-narrowed-type issues across the
+  package; `ruff check`, `ruff format --check`, and `mypy` all pass.
+
+### Security
+
+- `.gitignore` no longer relies on a blanket `*.csv` rule that silently hid
+  files; credential files used by examples are ignored by explicit path.
+- `evalring check` reports which variable supplied the API key without printing
+  the key, and a test asserts the key never reaches stdout.
+
+[Unreleased]: https://github.com/CoronRing/EvalRing/compare/v0.2.0...HEAD
+[0.2.0]: https://github.com/CoronRing/EvalRing/releases/tag/v0.2.0

evalring-0.2.0/CODE_OF_CONDUCT.md

@@ -0,0 +1,56 @@
+# Code of Conduct
+
+## Our pledge
+
+We want EvalRing to be a project where anyone can contribute without being
+harassed, belittled, or excluded. Everyone participating — in issues, pull
+requests, discussions, or any other project space — is expected to help keep it
+that way.
+
+## Expected behaviour
+
+- Be direct about code and considerate about people. Critique the change, not
+  the person who wrote it.
+- Assume competence and good faith. Contributors have context you do not.
+- Accept that reviewers and maintainers will sometimes say no, and give reasons
+  when you are the one saying it.
+- Respect that people work in different time zones, languages, and levels of
+  availability.
+
+## Unacceptable behaviour
+
+- Harassment, intimidation, or sustained disruption.
+- Insults or derogatory comments, including about someone's background,
+  identity, or experience level.
+- Publishing anyone's private information without their explicit permission.
+- Unwelcome sexual attention or imagery.
+- Any other conduct that would reasonably be considered inappropriate in a
+  professional setting.
+
+## A note on this project's subject matter
+
+Some examples in this repository involve suicide-risk classification and
+clinical data. Discuss that material with the seriousness it deserves. Do not
+post identifiable data from any dataset in an issue, a pull request, or a
+discussion.
+
+## Scope
+
+This applies in all project spaces — the repository, issues, pull requests,
+discussions — and when someone is representing the project publicly.
+
+## Reporting
+
+Report unacceptable behaviour by opening a
+[private security advisory](https://github.com/CoronRing/EvalRing/security/advisories/new),
+which is the available private channel, or by contacting a maintainer directly.
+Reports are handled confidentially and reviewed as promptly as we can manage.
+
+Maintainers may edit or remove contributions that violate this policy, and may
+temporarily or permanently ban anyone for behaviour they judge inappropriate,
+threatening, or harmful.
+
+## Attribution
+
+Adapted from the [Contributor Covenant](https://www.contributor-covenant.org),
+version 2.1.

evalring-0.2.0/CONTRIBUTING.md

@@ -0,0 +1,137 @@
+# Contributing to EvalRing
+
+Thanks for your interest. This document covers how to get set up, what the code
+should look like, and what a reviewable change looks like.
+
+## Getting set up
+
+EvalRing requires Python 3.10 or newer.
+
+```bash
+git clone https://github.com/CoronRing/EvalRing.git
+cd EvalRing
+
+python -m venv .venv
+source .venv/bin/activate        # Windows: .venv\Scripts\activate
+
+pip install -e ".[dev]"
+```
+
+`[dev]` pulls in the full runtime (`llm`, `viz`, `datagen`) plus pytest, ruff,
+mypy, and the build tooling.
+
+## The checks CI runs
+
+Run these before opening a pull request. They are exactly what
+[`.github/workflows/ci.yml`](.github/workflows/ci.yml) runs.
+
+```bash
+ruff check src tests          # lint
+ruff format --check src tests # formatting
+mypy                          # types
+pytest                        # tests
+```
+
+`ruff format src tests` applies the formatting rather than just checking it.
+
+### Tests must run offline
+
+The whole suite runs without an API key and without network access. Provider
+credentials are stripped from the environment by an autouse fixture in
+[`tests/conftest.py`](tests/conftest.py), and cache and run artifacts are
+redirected into a temporary directory.
+
+If you add a test that genuinely needs the network, mark it:
+
+```python
+@pytest.mark.network
+def test_fetches_the_live_model_catalogue() -> None: ...
+```
+
+Those tests are excluded with `pytest -m "not network"`.
+
+To test code that calls a model, subclass `BaseAgent` with a scripted response
+rather than mocking an SDK — see `ScriptedAgent` in
+[`tests/test_evaluator.py`](tests/test_evaluator.py).
+
+## Design principles
+
+**Push shared logic down into the framework, not up into each task.** If two
+evaluation tasks both need a behaviour, it belongs in `EvalRing/`, not copied
+into two scripts under `examples/`. Dataset-level and evaluator-level
+abstractions exist so a new task is a thin script, not a fork of the framework.
+
+**Never hard-code a provider.** All credentials and endpoints resolve through
+[`EvalRing/config.py`](src/EvalRing/config.py). A new backend adds an entry to
+`_CREDENTIAL_SOURCES`; it does not read `os.environ` directly and it never bakes
+in a base URL as a default argument.
+
+**The library logs, the entry point prints.** Modules under `EvalRing/` use
+`get_logger(__name__)` and never call `print()` or configure the root logger.
+Only `EvalRing/cli.py` and scripts under `examples/` call
+`configure_logging()`. The one exception is machine-readable output the user
+asked for, which goes to stdout via `print()` in the CLI.
+
+**Optional dependencies stay optional.** `import EvalRing` must succeed with
+only the core dependencies installed. Import `openai`, `litellm`, `matplotlib`,
+and `nest_asyncio` inside the function that needs them, and raise an
+`ImportError` naming the extra to install. CI enforces this in the
+`minimal-install` job.
+
+## Code style
+
+- Formatting and linting are ruff's job; do not hand-format around it.
+- Public functions, classes, and methods need a docstring with `Args:`,
+  `Returns:`, and `Raises:` sections where they apply. Docstrings render in
+  Pylance, so keep the argument names accurate.
+- Annotate public signatures. `EvalRing.config`, `EvalRing.logging_utils`, and
+  `EvalRing.cli` are checked under stricter mypy settings; new modules should
+  be added to that override list in `pyproject.toml` once they are fully typed.
+- Do not reference previous versions of the code in comments. Comments explain
+  what the code does now and why, not what it used to be.
+
+## Documentation is part of the change
+
+Docs ship with the code so that a reader — human or agent — can trust them
+without reading the source. A change that alters behaviour is not complete
+until the affected document is updated in the same pull request:
+
+| If you change | Update |
+| --- | --- |
+| Architecture, components, data flow | [`docs/DESIGN_SPEC.md`](docs/DESIGN_SPEC.md) and bump its version |
+| Environment variables, provider resolution | [`docs/CONFIGURATION.md`](docs/CONFIGURATION.md) |
+| The public Python API | [`docs/API_REFERENCE.md`](docs/API_REFERENCE.md) |
+| CLI flags or subcommands | [`docs/CLI.md`](docs/CLI.md) |
+| How runs are operated | [`docs/USAGE.md`](docs/USAGE.md) |
+| Anything user-visible | [`CHANGELOG.md`](CHANGELOG.md) under `Unreleased` |
+
+[`AGENTS.md`](AGENTS.md) is the entry point for coding agents working in this
+repository; keep it pointing at the right documents.
+
+## Data and secrets
+
+- Never commit an API key. Credentials come from the environment or a local
+  `.env`, both of which are gitignored. See [`.env.example`](.env.example).
+- Never commit a dataset you do not have redistribution rights to. Datasets
+  used by examples are documented in [`docs/DATA.md`](docs/DATA.md) with their
+  source and licence; add an entry there before adding data.
+- The suicide-detection and clinical examples involve sensitive material. Read
+  [`docs/DATA.md`](docs/DATA.md) before working on them.
+
+## Pull requests
+
+1. Branch from `main`.
+2. Keep the change focused; unrelated formatting churn makes review harder.
+3. Add or update tests covering the behaviour you changed.
+4. Update the documents listed above.
+5. Make sure the four checks pass locally.
+
+Commits do not need to follow a strict convention, but a subject line that
+names the behaviour that changed helps the changelog.
+
+## Reporting bugs and requesting features
+
+Open an issue using the templates under
+[`.github/ISSUE_TEMPLATE`](.github/ISSUE_TEMPLATE). For anything with security
+implications, follow [`SECURITY.md`](SECURITY.md) instead of opening a public
+issue.

evalring-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 CoronRing
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

evalring-0.2.0/MANIFEST.in

@@ -0,0 +1,22 @@
+# The sdist carries the documentation alongside the code, so anyone reading an
+# unpacked release — a person or an agent — has the authoritative reference
+# without needing the repository.
+include LICENSE
+include README.md
+include CHANGELOG.md
+include CONTRIBUTING.md
+include CODE_OF_CONDUCT.md
+include SECURITY.md
+include AGENTS.md
+include .env.example
+include src/EvalRing/py.typed
+recursive-include docs *.md
+
+# Not part of a release: runnable examples, the test suite, and the vendored
+# upstream benchmark checkout.
+prune examples
+prune tests
+prune CounselBench
+prune .github
+prune .agent_temp
+global-exclude __pycache__ *.py[cod] *.so .DS_Store