valcore 0.2.2__tar.gz → 0.2.4__tar.gz

valcore-0.2.4/PKG-INFO

@@ -0,0 +1,493 @@
+Metadata-Version: 2.5
+Name: valcore
+Version: 0.2.4
+Summary: A small, self-contained local tool for developing, improving, and running agentic evaluations.
+Author-email: Duncan McKinnon <dkmckinnon@outlook.com>
+License-Expression: Apache-2.0
+License-File: LICENSE
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: Microsoft :: Windows
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.11
+Requires-Dist: click
+Requires-Dist: fastapi
+Requires-Dist: logfire[fastapi,system-metrics]<5,>=4.39
+Requires-Dist: pydantic-ai-harness[code-mode]<1,>=0.12
+Requires-Dist: pydantic-ai<3,>=2.19
+Requires-Dist: pydantic-settings
+Requires-Dist: python-multipart
+Requires-Dist: sqlmodel
+Requires-Dist: uvicorn[standard]
+Description-Content-Type: text/markdown
+
+<p align="center">
+  <img src="docs/img/logo.png" alt="valcore" width="200" />
+</p>
+
+<p align="center">
+  <a href="https://github.com/duncankmckinnon/valcore/actions/workflows/test.yml">
+    <img src="https://github.com/duncankmckinnon/valcore/actions/workflows/test.yml/badge.svg" alt="CI">
+  </a>
+  <a href="https://codecov.io/gh/duncankmckinnon/valcore">
+    <img src="https://codecov.io/gh/duncankmckinnon/valcore/graph/badge.svg" alt="codecov">
+  </a>
+  <a href="https://pypi.org/project/valcore/">
+    <img src="https://img.shields.io/pypi/v/valcore" alt="PyPI">
+  </a>
+  <a href="https://github.com/duncankmckinnon/homebrew-tap">
+    <img src="https://img.shields.io/badge/homebrew-duncankmckinnon%2Ftap-orange?logo=homebrew" alt="Homebrew">
+  </a>
+  <a href="https://pypi.org/project/valcore/">
+    <img src="https://img.shields.io/pypi/pyversions/valcore" alt="Python">
+  </a>
+  <a href="https://github.com/duncankmckinnon/valcore/blob/main/LICENSE">
+    <img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License: Apache 2.0">
+  </a>
+</p>
+
+<!--
+Absolute raw.githubusercontent URL rather than a relative path: this README is the
+PyPI long_description, and PyPI resolves relative paths against pypi.org, so a
+relative image renders broken there. The URL is pinned to main, so it only resolves
+once this lands on the default branch.
+-->
+
+A small, self-contained local tool for developing, improving, and running agentic
+evaluations. Author evaluators in a web UI, run them over datasets from the command
+line, and gate CI on their accuracy.
+
+## Install
+
+On macOS or Linux, with Homebrew:
+
+```bash
+brew install duncankmckinnon/tap/valcore
+```
+
+On macOS, Linux, or Windows, as a `uv` tool:
+
+```bash
+uv tool install valcore
+```
+
+In PowerShell, the same `uv tool install valcore` command installs the Windows
+entrypoint. Either route gives you a `valcore` command on your `PATH`; run
+`valcore serve` to open the interface.
+
+## Web UI
+
+`valcore serve` opens a dark-themed web UI with five surfaces:
+
+- **Overview** — the landing page, summarizing what you have and pointing to the next
+  step.
+- **Evaluators** — author, version, and validate LLM-as-judge evaluators.
+- **Datasets** — build and edit the datasets evaluators run over, by hand, generated
+  from a description, or pulled from a Logfire query.
+- **Runs** — inspect completed runs, their metrics, and per-row scores, and compare runs
+  against each other.
+- **Docs** — opens the canonical guides and CLI reference on
+  [e-valcore.com](https://e-valcore.com/docs).
+
+The website docs explain the concepts the other four surfaces assume — label spaces,
+frozen versions, run kinds, agreement metrics, installation, credentials, portable
+packages, CI, and Logfire. The app links there instead of maintaining a second copy.
+
+## Quickstart
+
+Store your gateway API key, then start the app:
+
+```bash
+valcore config set-key sk-...
+valcore serve
+```
+
+`serve` starts the web UI and API on <http://127.0.0.1:8000> and opens a browser
+(pass `--no-browser` to skip that, or `--port` to bind elsewhere). Author evaluators
+and datasets in the UI, then drive runs from the command line. Both can be written by
+hand or generated from a description; a generated result is an editable draft either way.
+
+From there, **Docs** in the app opens the website reference for the full workflow —
+authoring a judge, getting labeled rows, and reading what a run measured.
+
+## Seeding one from the other
+
+An evaluator and a dataset have to agree on columns, so rather than retype that shape you
+can seed either one from the other and let the model fill in content you describe.
+
+Generate a dataset from an evaluator version and the dataset always gets that version's
+required columns; you can add extra columns by naming them, and per-column notes say what
+each should contain. Suggested labels are optional — ask for them when you want the model
+to propose ground truth, and the label space comes from the evaluator.
+
+Generate an evaluator from a dataset and it is drafted against that dataset's columns,
+with per-column notes saying how each factors into the assessment. The result is an
+editable draft, not a saved version, so you review and adjust it before keeping it.
+
+A dataset needs no labels to be scored: an ordinary run just records the judge's output.
+Labels are only required for a `validation` run, which compares the judge against them to
+measure agreement.
+
+## Models and the gateway
+
+valcore reaches hosted models through the
+[Pydantic AI Gateway](https://ai.pydantic.dev/gateway/). That is the only hosted route:
+there is no direct-to-provider client and no per-provider API key, so a gateway key is
+required before anything that calls a hosted model will run.
+
+Hosted model strings are `gateway/<provider>:<model>`:
+
+```
+gateway/anthropic:claude-sonnet-5      # the default
+gateway/anthropic:claude-opus-4-5
+gateway/anthropic:claude-haiku-4-5
+gateway/openai:gpt-5
+gateway/google:gemini-2.5-pro
+```
+
+Valid providers are `anthropic`, `openai`, `google`, `google-cloud`, `bedrock`, and
+`groq`. A string that does not match this shape is rejected before any request is made,
+so a bare `claude-sonnet-5` fails fast with a clear error rather than at call time.
+
+The key is stored in `~/.valcore/config.toml` (mode `0600`) and exported as
+`PYDANTIC_AI_GATEWAY_API_KEY` when a command runs. An already-exported environment
+variable always wins over the stored key, which is what you want in CI:
+
+```bash
+export PYDANTIC_AI_GATEWAY_API_KEY=sk-...
+```
+
+Run `valcore config set-key` with no argument to be prompted without echoing the key.
+
+Every key in `config.toml` can also be set generically. `valcore config set <key> <value>`
+validates before writing (a bad model string, an unknown local CLI, or a non-numeric
+`concurrency` is refused), and `valcore config unset <key>` removes one. The named
+`set-*` commands remain the way to enter a secret without it appearing in shell history.
+
+A local model — `local/claude`, `local/codex`, or `local/cursor` — reuses an already
+logged-in CLI on this machine instead, and needs no gateway key. There is no model name
+after the CLI: each route runs that CLI's own binary (`claude`, `codex`, or
+`cursor-agent`), which must be on `PATH` and already authenticated, and it answers with
+whichever model it is configured to use.
+
+Pick one under **Model Selection** in Settings, or from the command line:
+
+```bash
+valcore config set local_cli_default claude   # or codex, cursor
+valcore config unset local_cli_default        # back to the gateway
+```
+
+Either way it is stored as `local_cli_default` in `config.toml`; a version can still name a
+local route explicitly as its `model`. Overview's default-model card shows whichever model
+actually resolves.
+
+Override the default model, highest precedence first: an explicit argument,
+`VALCORE_DEFAULT_MODEL`, `local_cli_default` in `config.toml`, `model` in `config.toml`,
+then the built-in default.
+
+> **On other providers.** Routing every hosted call through one gateway keeps model access
+> to a single credential and a single validated string format. It also means valcore
+> inherits whatever the gateway supports and nothing else — the local CLI routes above are
+> the one way around that, and they borrow a CLI's existing login rather than adding a
+> provider. Provider routing is confined to one module, so widening this later — direct
+> provider clients, a self-hosted or OpenAI-compatible endpoint — is a change to that
+> resolution layer and the config schema rather than a change to how evaluators, datasets,
+> or runs work.
+
+## Setup
+
+`valcore serve` shows a setup card on the Overview page listing each key valcore knows
+about and whether it is currently set. Open **Settings** to paste or update keys; they
+are stored in `~/.valcore/config.toml` and never shown back (a set field is masked).
+The same values can still be set from the CLI:
+
+```bash
+valcore config set-key                  # required: runs and generation
+valcore config set-logfire-token        # optional: traces to your valcore Logfire project
+valcore config set-logfire-read-key     # optional: query traces and hosted datasets in the project you sample from
+valcore config set-logfire-write-key    # optional: push datasets to your valcore Logfire project
+valcore config set-logfire-key          # optional: one API key stored as both (same project only)
+valcore config set-logfire-explore-url  # optional fallback if the read key cannot resolve the project
+```
+
+Without the gateway key, generation and runs are unavailable, and the UI shows why. Manual
+authoring, dataset upload, editing, hand-labeling, and every export still work with no key
+configured at all.
+
+## Commands
+
+The full CLI reference — with option-by-option detail, examples, exit codes, and
+configuration keys — ships as part of the [agent skill](#agent-skills) at
+[`reference.md`](src/valcore/skills/use-valcore/reference.md); it is this table's
+source of truth (`scripts/sync_command_table.py --check` fails CI if the two drift).
+
+<!-- COMMANDS:START -->
+| Command | What it does |
+| --- | --- |
+| `valcore serve` | Serve the web UI and API (`--port`, `--host`, `--no-browser`). |
+| `valcore list <evaluators\|datasets\|runs\|agents\|derivations>` | List resources as a table or, with `--json`, as JSON. |
+| `valcore run agent <agent>` | Run an agent over a dataset, one row, ad-hoc inputs, or a literal prompt. |
+| `valcore run evaluator <evaluator> --dataset <dataset>` | Run an evaluator version over a dataset. |
+| `valcore run experiment <evaluator> --dataset <dataset>` | Run an evaluator through `pydantic_evals.Dataset.evaluate`. |
+| `valcore export <evaluator>` | Export an evaluator (and, with `--dataset`, a dataset) as a Python script or, with `--format json`, a portable eval package. |
+| `valcore import <file>` | Import a JSON eval package back into the local database. |
+| `valcore agent derivation list` | List saved and staged response derivations. |
+| `valcore agent derivation save <ref>` | Accept a staged derivation. |
+| `valcore agent derivation discard <ref>` | Discard a derivation. |
+| `valcore agent prompt-sync status <agent>` | Inspect local agent text and the configured Logfire project's latest variable versions. |
+| `valcore agent prompt-sync link <agent> --initial local\|remote` | Link an agent, choosing the initial text source. |
+| `valcore agent prompt-sync pull <agent>` | Pull remote text into a new local agent version, with confirmation. |
+| `valcore agent prompt-sync push <agent>` | Push local text into new Logfire variable versions, with confirmation. |
+| `valcore agent prompt-sync resolve <agent> --choice local\|remote --field FIELD` | Show a three-way diff and explicitly resolve selected conflicts. |
+| `valcore agent prompt-sync unlink <agent>` | Remove the local sync link. |
+| `valcore agent import <file>` | Import a YAML or JSON AgentSpec and its valcore binding. |
+| `valcore agent export <agent>` | Export an agent version as a YAML AgentSpec with its valcore binding. |
+| `valcore config set <key> <value>` | Set any config key, including `model`, `local_cli_default`, `port`, `concurrency`, and `db_path`. |
+| `valcore config unset <key>` | Remove any config key. |
+| `valcore config set-key [KEY]` | Store the gateway API key in the config file. |
+| `valcore config set-logfire-token [TOKEN]` | Store the Logfire tracing token in the config file. |
+| `valcore config set-logfire-key [KEY]` | Store one Logfire API key as both the read and write keys. |
+| `valcore config set-logfire-read-key [KEY]` | Store the Logfire read key (query traces and hosted datasets in the source project). |
+| `valcore config set-logfire-write-key [KEY]` | Store the Logfire write key (push datasets to the valcore project). |
+| `valcore config set-logfire-explore-url [URL]` | Optional fallback SQL Workbench URL if the read key cannot resolve the project. |
+| `valcore config get` | Show the current config (the key is masked unless `--show-key`). |
+| `valcore config path` | Print the path to the config file. |
+| `valcore config edit` | Open the config file in `$EDITOR`. |
+| `valcore logfire pull` | Create a dataset from a Logfire SQL query (`--sql` or `--sql-file`, `--name`, `--count`). |
+| `valcore logfire list` | List hosted datasets in the source Logfire project. |
+| `valcore logfire fetch <name>` | Create a local dataset from a hosted Logfire dataset. |
+| `valcore logfire push <dataset>` | Push a dataset to Logfire's hosted dataset store. |
+| `valcore skills install` | Install the bundled agent skills (`--claude`, `--copilot`, …). |
+| `valcore skills list` | Show the bundled skills and where each is installed. |
+| `valcore skills uninstall` | Remove the bundled skills from the selected directories. |
+| `valcore version` | Print the installed valcore version. |
+<!-- COMMANDS:END -->
+
+Evaluators, versions, and datasets are addressable by name or by a unique id prefix;
+an ambiguous value is an error that lists the candidates. Pass `--db PATH` on the group
+to point at a SQLite database other than the default under `~/.valcore`.
+
+`run` accepts `--version` (defaults to the active version), `--kind`, `--concurrency`,
+`--watch` (one line per completed row), `--json`, and `--min-accuracy`. Progress goes to
+stderr and results go to stdout, so redirecting stdout yields clean JSON. The CLI talks
+to SQLite directly, so `run` works whether or not `serve` is up.
+
+## Portable eval packages
+
+Both an evaluator and a dataset export two ways, as **code** or as **JSON**:
+
+| | Code | JSON |
+| --- | --- | --- |
+| Evaluator | standalone Python script | `pydantic_ai` `AgentSpec` |
+| Dataset | module that builds a `pydantic_evals.Dataset` | `pydantic_evals` `Dataset` |
+
+The JSON forms combine into an **eval package**: a `pydantic_evals` dataset plus a
+`pydantic_ai` `AgentSpec`, in one file by default or two with `--split`. Neither half is
+invented here — each is the serialization its own framework already defines — and a small
+`valcore` block beside them carries the prompt template, required columns, score field, and
+tool names that neither foreign format has a place for.
+
+```bash
+valcore export my-judge                                  # standalone Python script
+valcore export my-judge --format json -o my-judge.json
+valcore export --dataset my-data --format json -o my-data.json
+valcore export my-judge --dataset my-data --format json --split -o pkg.json
+valcore import my-judge.json
+```
+
+`valcore export my-judge` with no new flags still emits exactly the Python script it always
+has. `--format json` emits the package instead, `--dataset` folds a dataset into either form,
+and `--split` writes `pkg.agent.json` and `pkg.dataset.json` side by side rather than one
+bundle. `import` reads the JSON form back into your local database; a `.py` export is not
+importable.
+
+### Running a package
+
+A `valcore_judge.py` companion module ships beside the JSON and needs no valcore install — it
+imports only stdlib `json` and pydantic, rebuilding the agent with `Agent.from_spec`. Register
+it as a custom evaluator type and the dataset runs under `pydantic_evals`:
+
+```python
+from pydantic_evals import Dataset
+
+from valcore_judge import ValcoreJudge
+
+dataset = Dataset.from_file("my-data.json", custom_evaluator_types=[ValcoreJudge])
+report = dataset.evaluate_sync(task)
+```
+
+### What the formats can and cannot do
+
+Three limitations are worth stating plainly:
+
+1. Reading a **bundled** package needs `valcore_judge.py`, because `pydantic_evals.Dataset`
+   forbids unknown top-level keys and so refuses the bundle's `agent` and `valcore` blocks on
+   its own.
+2. A dataset exported with an evaluator names `ValcoreJudge` in its `evaluators`, so a bare
+   `Dataset.from_file()` — with no `custom_evaluator_types` — cannot read it either.
+3. `AgentSpec` has no `tools` field and silently ignores unknown keys, so
+   `Agent.from_spec(AgentSpec.from_file("pkg.agent.json"))` builds a **working agent with zero
+   tools** without complaint — the tool names live in the `valcore` block, which `AgentSpec`
+   drops on load. Use `valcore_judge.py`, which restores them from source inlined into the
+   module. If you load the bare spec and wonder why the judge behaves differently, this is why.
+
+JSON is the only config format: the web UI validates and previews a package client-side with
+its built-in `JSON.parse`, adding no dependency to do it.
+
+## Agent skills
+
+valcore ships a skill document that teaches a coding agent how to drive it — the data
+model, the author/validate/run/export loop, the evaluator-dataset compatibility rules,
+and a full CLI reference. Install it into whichever agent you use:
+
+```bash
+valcore skills install --claude          # ./.claude/skills/
+valcore skills install --claude --copilot
+valcore skills install                   # ./.agents/skills/, discoverable by any client
+valcore skills install --claude --global # ~/.claude/skills/
+```
+
+| Flag | Destination |
+| --- | --- |
+| *(none)* or `--agents` | `.agents/skills/` |
+| `--claude` | `.claude/skills/` |
+| `--copilot` | `.github/skills/` |
+| `--all` | all of the above |
+
+Flags are additive and nothing is implicit — `--claude --copilot` writes exactly those
+two directories and leaves `.agents/` alone. Add `--global` for home-level directories
+instead of the current repository.
+
+Copying is the default: an already-identical skill is skipped, and one you have edited
+prompts before being overwritten (`--force` to skip the prompt). Use `--symlink` to link
+to the packaged copy instead, so upgrading valcore upgrades the installed skill.
+
+Run `valcore skills list` to see what is bundled and where each copy currently lives.
+
+## Using valcore in CI
+
+`run evaluator --json` emits a single object with run metadata, metrics, and per-row scores, and
+`--min-accuracy` turns a validation run into a pass/fail gate:
+
+```bash
+valcore run evaluator my-evaluator --dataset my-dataset \
+  --kind validation \
+  --min-accuracy 0.9 \
+  --json > run.json
+```
+
+Exit codes:
+
+| Exit | Meaning |
+| --- | --- |
+| `0` | The run finished and, if `--min-accuracy` was set, accuracy met the threshold. |
+| `1` | The run failed, or a domain error occurred (printed as `error: <message>` on stderr). |
+| `2` | Accuracy fell below `--min-accuracy`. |
+
+`--min-accuracy` requires a categorical accuracy metric; numeric or unlabeled runs have
+no accuracy and error rather than silently passing.
+
+## Logfire
+
+`logfire` is an optional extra — install it to get traces:
+
+```bash
+uv tool install 'valcore[logfire]'
+```
+
+With a Logfire token configured (see [Setup](#setup)), each `valcore run` opens a
+`valcore.run` span carrying the run's agent or evaluator version, dataset, and concurrency,
+with a child span per row; on close, the run span records its status and available metrics as
+attributes, so a Logfire query can filter evaluation runs by accuracy directly.
+The [Pydantic AI Gateway](https://ai.pydantic.dev/gateway/) already reports the LLM calls
+themselves — valcore adds only the surrounding run and row context around them, and
+deliberately does not re-report the calls, which would double-count tokens and cost.
+
+The Settings page can also opt the browser UI into Logfire frontend observability. Create
+a frontend application in Logfire, then paste its generated trace URL and restricted
+public token into **Logfire Frontend Observability**. Browser tracing starts after the
+next page load. Session replay is a separate, off-by-default Early Access option; when
+enabled, valcore records UI text and input values outside the blocked Settings page and
+disables console capture. See Logfire's
+[frontend observability guide](https://pydantic.dev/logfire/observe/frontend/) for the
+Logfire-side setup.
+
+Agent `instructions` and input templates can also be synced explicitly with ordinary Logfire
+managed variables (not Prompt Management objects). Runs use the local agent version and make
+no prompt-sync calls to Logfire. See
+[Agent prompt sync with Logfire](docs/agent-logfire-prompt-sync.md) for the required
+`project:read_variables` and `project:write_variables` key, the conflict flow, and limits.
+
+`valcore run experiment <evaluator> --dataset <dataset>` runs the same evaluation through
+`pydantic_evals.Dataset.evaluate` instead of the evaluator runner, so it also appears in
+Logfire's experiments view on the valcore tracing project. It persists a run the same way
+`run evaluator` does, so it shows up on the Runs page too. Unlike that runner, it cannot be
+cancelled, because `Dataset.evaluate` has no cancellation.
+
+`valcore logfire pull --sql '…' --name traces --count 20` runs that SQL against Logfire,
+nests child spans that the query actually returned, samples top-level entries, and stores
+them as a local dataset. Child spans become a JSON `children` column on the parent row.
+The read key needs `project:read` for the query and `project:read_datasets` to list or
+fetch hosted datasets. SQL Workbench and live traces open from that key's project. A
+stored Explore URL is only used if that lookup fails:
+
+```bash
+valcore config set-logfire-explore-url https://logfire-us.pydantic.dev/org/project/explore
+```
+
+`valcore logfire list` shows hosted datasets in the source project.
+`valcore logfire fetch qa-set` copies one into the local store (optionally `--name` /
+`--description` for the local copy). The Logfire tab in the Datasets UI offers the same
+SQL and hosted sources.
+
+A dataset created from either Logfire source can pull in more data later, from its detail
+page: a SQL-pulled dataset gets a "Pull more from Logfire" action that repeats the stored
+query (or an edited one) and appends the result; a hosted-fetch dataset gets a "Sync from
+Logfire" action that refetches the same hosted dataset and appends only the cases not
+already present by content, leaving existing rows (including any local relabeling) untouched.
+Both are UI-only for now, with no CLI equivalent.
+
+`valcore logfire push <dataset>` publishes a dataset to Logfire's hosted dataset store on
+the valcore project (the same project experiment runs appear in). The Datasets page and a
+dataset's detail view open that project's evals pages from the write key. The write key
+needs `project:read_datasets` and `project:write_datasets`.
+
+## `~/.valcore`
+
+All state lives under the current user's `~/.valcore` directory. In PowerShell this is
+`$HOME\.valcore`, normally `C:\Users\<name>\.valcore`. Set `VALCORE_HOME` to relocate it.
+
+On macOS and Linux, valcore creates the directory with mode `0700` and
+`config.toml` with mode `0600`. On Windows, access is governed by the current user's
+filesystem ACLs.
+
+```
+~/.valcore/
+  config.toml                    gateway key + defaults
+  valcore.db             SQLite (plus -wal, -shm)
+  logs/                    serve logs
+```
+
+## Development
+
+```bash
+uv sync                    # install dependencies into a local venv
+uv run pytest              # run the test suite
+```
+
+The web UI is a Vite + React SPA under `web/`:
+
+```bash
+cd web
+npm install
+npm run dev                # Vite dev server, proxying the API
+```
+
+To build the SPA into the wheel, run `npm run build` and copy `web/dist/` into
+`src/valcore/web_dist/` before `uv build`; the release workflow does this automatically
+on a `v*` tag.