valcore 0.1.2__tar.gz → 0.2.0__tar.gz

valcore-0.2.0/.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,24 @@
+---
+name: Bug report
+about: Something in valcore isn't working as expected
+title: ""
+labels: bug
+---
+
+**What happened?**
+
+**What did you expect to happen instead?**
+
+**Steps to reproduce**
+
+1.
+2.
+3.
+
+**Environment**
+
+- valcore version (`valcore --version`):
+- OS:
+- Install method: Homebrew / `uv tool install` / other
+
+**Anything else?**

valcore-0.2.0/.github/pull_request_template.md

@@ -0,0 +1,11 @@
+## Summary
+
+-
+
+Closes #
+
+## Test plan
+
+- [ ] `uv run pytest`
+- [ ] `cd web && npx vitest run`
+- [ ] `cd web && npx tsc --noEmit`

valcore-0.2.0/.github/workflows/test.yml

@@ -0,0 +1,89 @@
+name: test
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+concurrency:
+  group: test-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v5
+      - uses: actions/setup-python@v6
+        with:
+          python-version: "3.12"
+      - uses: astral-sh/setup-uv@v6
+        with:
+          enable-cache: true
+      - name: Install dependencies
+        run: uv sync --all-extras
+      # Runs the same hooks contributors get locally, so CI and a clean
+      # `git commit` can never disagree about formatting.
+      - name: Run pre-commit
+        run: uv run pre-commit run --all-files --show-diff-on-failure
+
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.11", "3.12", "3.13", "3.14"]
+    steps:
+      - uses: actions/checkout@v5
+      - uses: astral-sh/setup-uv@v6
+        with:
+          enable-cache: true
+      - name: Install dependencies on Python ${{ matrix.python-version }}
+        run: uv sync --python ${{ matrix.python-version }}
+      - name: Run tests
+        run: uv run --python ${{ matrix.python-version }} pytest -q --cov=valcore --cov-report=xml
+
+      # One upload is enough -- the matrix runs the same suite, so the other
+      # versions would only add duplicate reports for Codecov to merge.
+      # fail_ci_if_error stays false so a missing or rotated CODECOV_TOKEN
+      # degrades the badge rather than failing an otherwise-green build.
+      - name: Upload coverage to Codecov
+        if: matrix.python-version == '3.12'
+        uses: codecov/codecov-action@v5
+        with:
+          token: ${{ secrets.CODECOV_TOKEN }}
+          files: coverage.xml
+          fail_ci_if_error: false
+
+  windows:
+    runs-on: windows-latest
+    steps:
+      - uses: actions/checkout@v5
+      - uses: astral-sh/setup-uv@v6
+        with:
+          enable-cache: true
+      - name: Install dependencies on Python 3.12
+        run: uv sync --python 3.12
+      - name: Run tests on Windows
+        run: uv run --python 3.12 pytest -q
+
+  web:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v5
+      - uses: actions/setup-node@v5
+        with:
+          node-version: "22"
+          cache: npm
+          cache-dependency-path: web/package-lock.json
+      - name: Install web dependencies
+        run: npm ci
+        working-directory: web
+      # `npm run build` runs tsc --noEmit before vite, so this is the
+      # typecheck as well as the build.
+      - name: Build the SPA
+        run: npm run build
+        working-directory: web
+      - name: Run web tests
+        run: npx vitest run
+        working-directory: web

valcore-0.2.0/.gitignore

@@ -0,0 +1,237 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[codz]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#   Usually these files are written by a python script from a template
+#   before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py.cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+# Pipfile.lock
+
+# UV
+#   Similar to Pipfile.lock, it is generally recommended to include uv.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+# uv.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+# poetry.lock
+# poetry.toml
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#   pdm recommends including project-wide configuration in pdm.toml, but excluding .pdm-python.
+#   https://pdm-project.org/en/latest/usage/project/#working-with-version-control
+# pdm.lock
+# pdm.toml
+.pdm-python
+.pdm-build/
+
+# pixi
+#   Similar to Pipfile.lock, it is generally recommended to include pixi.lock in version control.
+# pixi.lock
+#   Pixi creates a virtual environment in the .pixi directory, just like venv module creates one
+#   in the .venv directory. It is recommended not to include this directory in version control.
+.pixi
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# Redis
+*.rdb
+*.aof
+*.pid
+
+# RabbitMQ
+mnesia/
+rabbitmq/
+rabbitmq-data/
+
+# ActiveMQ
+activemq-data/
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.envrc
+.venv
+.worktrees/
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#   JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#   be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#   and can be added to the global gitignore or merged into this file.  For a more nuclear
+#   option (not recommended) you can uncomment the following to ignore the entire idea folder.
+# .idea/
+
+# Abstra
+#   Abstra is an AI-powered process automation framework.
+#   Ignore directories containing user credentials, local state, and settings.
+#   Learn more at https://abstra.io/docs
+.abstra/
+
+# Visual Studio Code
+#   Visual Studio Code specific template is maintained in a separate VisualStudioCode.gitignore
+#   that can be found at https://github.com/github/gitignore/blob/main/Global/VisualStudioCode.gitignore
+#   and can be added to the global gitignore or merged into this file. However, if you prefer,
+#   you could uncomment the following to ignore the entire vscode folder
+# .vscode/
+# Temporary file for partial code execution
+tempCodeRunnerFile.py
+
+# Ruff stuff:
+.ruff_cache/
+
+# PyPI configuration file
+.pypirc
+
+# Marimo
+marimo/_static/
+marimo/_lsp/
+__marimo__/
+
+# Streamlit
+.streamlit/secrets.toml
+.serena/*
+.claude/*
+.workbench/*
+.agents/*
+
+docs/superpowers/*
+
+# Written at build time by hatch-vcs from the git tag; never committed.
+src/valcore/_version.py
+
+# valcore local state
+valcore.db
+valcore.db-shm
+valcore.db-wal
+valcore.db*
+web/dist/
+
+.superpowers/

valcore-0.2.0/AGENTS.md

@@ -0,0 +1,167 @@
+# AGENTS.md
+
+Instructions and orientation for a coding agent working in this repository.
+For contribution mechanics written for a human, see
+[CONTRIBUTING.md](CONTRIBUTING.md); for user-facing behavior, see
+[README.md](README.md).
+
+## Cheat sheet
+
+```bash
+uv sync --all-extras              # install backend deps (dev group + extras)
+cd web && npm install             # install frontend deps
+
+uv run pytest                     # backend tests
+cd web && npx vitest run          # frontend tests
+cd web && npx tsc --noEmit        # frontend typecheck
+
+uv run pre-commit run --all-files # ruff check --fix, ruff format, misc hygiene hooks
+uv run valcore serve --no-browser # run the API + built-in UI on :8000
+```
+
+- Backend source: `src/valcore/`. Frontend source: `web/src/` (Vite + React SPA,
+  proxies `/api` to the backend in dev).
+- One backend test file per source module — `src/valcore/store.py` is tested by
+  `tests/test_store.py`, `src/valcore/api/routes/datasets.py` by
+  `tests/test_api_datasets.py`, and so on. Add tests to the matching file.
+- This project practices test-first development: write the failing test, watch
+  it fail for the right reason, then write the minimal code to pass it.
+- If you touch a CLI command, run `python scripts/sync_command_table.py`
+  afterward — it syncs README's command table from the canonical one in
+  `src/valcore/skills/use-valcore/reference.md`, and `tests/test_docs.py`
+  enforces the two stay in sync.
+- Full command details and PR expectations are in
+  [CONTRIBUTING.md](CONTRIBUTING.md).
+
+## What valcore is
+
+valcore is a small, local-first tool for developing, improving, and running
+**agentic evaluations**: agent evaluators that score the outputs of
+some other agentic system against a dataset of examples, with a label space
+users define. The governing idea is that building a good evaluator is
+an iterative loop — author a judge, get labeled data for it to run against,
+run it, look at where it disagrees with ground truth, adjust the judge or the
+data, repeat — and valcore's job is to make every step of that loop fast and
+inspectable without requiring a hosted backend. Everything valcore itself
+persists (evaluators, versions, datasets, rows, runs) lives in a single local
+SQLite database at `~/.valcore/valcore.db`; nothing about authoring, editing,
+or running an evaluation requires a network call. The two places state leaves
+the machine are opt-in: a hosted model call through the Pydantic AI Gateway
+(or a local CLI, see below, which needs neither), and Logfire, described in
+its own section.
+
+## Three things this codebase calls "agents"
+
+The word "agent" refers to three distinct things depending on which part of
+the codebase you're in. Keeping them apart matters — conflating them is the
+easiest way to misread this code.
+
+### 1. Local CLI agents (`src/valcore/local_cli/`)
+
+valcore reaches hosted models exclusively through the Pydantic AI Gateway
+(model strings shaped `gateway/<provider>:<model>`), and that gateway needs a
+key. `local_cli/` is the alternative that needs no key at all: it wraps an
+already-installed, already-authenticated coding-agent CLI (`claude`, `codex`,
+or `cursor-agent`) as a `pydantic_ai.Model`, selected with a `local/<cli>`
+model string. `local_cli/bridge_model.py` defines the adapter protocol and the
+`CliBridgeModel` that shells out to the CLI's own binary; one adapter module
+per CLI (`claude_adapter.py`, `codex_adapter.py`, `cursor_adapter.py`) handles
+that tool's specific invocation and output parsing.
+`local_cli/__init__.py:resolve_model` is the single seam every other module
+routes a model string through — `factory.py`, `generator.py`, and
+`datagen.py` never talk to `Agent(...)` directly with a raw string. A
+`local/<cli>` route and a `gateway/...` route are otherwise interchangeable
+everywhere in valcore: dataset generation, evaluator generation, and judging
+all work the same way regardless of which one is configured.
+
+### 2. Evaluators as judge-agents (`factory.py`, `runner.py`, `experiment.py`)
+
+What valcore actually builds and versions is itself an agent: an
+`EvaluatorVersion` (instructions, prompt template, output schema, optional
+tools and harness capabilities) is a stored, immutable-once-frozen spec, and
+`factory.py` turns one into a live `pydantic_ai.Agent` that judges dataset
+rows and produces a structured score. `tools.py` and `capabilities.py` are
+what an evaluator agent can be given: deterministic row-inspection tools and
+the harness capability registry, respectively — both optional, and both
+resolved at the same seam local models are.
+
+Two engines run that agent over a dataset: `runner.py` is the primary one
+(bounded concurrency, per-row persistence as each result lands, cancellable),
+and `experiment.py` is a second path through
+`pydantic_evals.Dataset.evaluate`, used when you want the run to also appear
+in Logfire's experiments view. Both persist a `Run` the same way. `metrics.py`
+scores agreement between the judge's output and a row's label — pure
+functions, no I/O — and `schema_migration.py` handles what happens to
+existing labels when a dataset's shape changes underneath a running
+evaluator.
+
+### 3. The bundled valcore skill (`src/valcore/skills/`, installed via `valcore skills`)
+
+This direction runs the other way: it is not valcore using an agent, it is
+valcore teaching an *external* coding agent (Claude Code, Copilot, or any
+client that reads `.agents/skills/`) how to drive valcore as a tool — the
+data model, the author/validate/run/export loop, evaluator-dataset
+compatibility rules, and a full CLI reference. `cli/skills.py` implements
+`valcore skills install/list`. If you're an agent reading this file to work
+*on* valcore's own source, this is a different document from the one you
+want — see `src/valcore/skills/use-valcore/reference.md`, which is written
+for an agent operating valcore, not developing it.
+
+## Logfire as external persistence
+
+Logfire is optional and strictly outside valcore's own source of truth — the
+local SQLite database is authoritative with or without it — but it is where
+state actually leaves the local machine in both directions, not just one:
+
+- **Out**: `tracing.py` is the only module that talks to Logfire's
+  instrumentation surface. With a token configured, each `valcore run` opens
+  a `valcore.run` span carrying the evaluator version, dataset, and
+  concurrency, with a `valcore.score_row` child span per row; the run span
+  records its final status and agreement metrics as attributes on close. This
+  is observability, not persistence in the sense of being read back — valcore
+  never queries these spans itself.
+- **In**: `logfire_pull.py` runs an arbitrary SQL query against a Logfire
+  project's traces, nests child spans under their parents, samples top-level
+  entries, and maps the result into dataset rows — this is how real
+  production agent traces become eval data. `logfire_io.py` handles Logfire's
+  separate *hosted dataset* store: fetching one into the local SQLite store,
+  pushing a local dataset up to it, and the "pull more" / "sync" actions that
+  extend an existing local dataset from its Logfire source without
+  disturbing rows already there (a SQL-sourced dataset repeats its stored
+  query; a hosted-fetch dataset unions in only content not already present).
+
+Three credentials, independently optional, cover these directions: a Logfire
+token (write, for tracing), a read key (query traces, list/fetch hosted
+datasets), and a write key (push datasets, separate scope from the token).
+
+## Module map
+
+Backend, `src/valcore/`:
+
+| Module | Role |
+| --- | --- |
+| `models.py` | SQLModel entities, enums, validation |
+| `store.py` | SQLite persistence — the local source of truth |
+| `settings.py` | Model string validation and resolution, env config |
+| `config.py` / `config_io.py` | `~/.valcore/config.toml` layer; eval-package JSON I/O |
+| `paths.py` | Filesystem layout under `~/.valcore` |
+| `factory.py` | Stored `EvaluatorVersion` → live `pydantic_ai.Agent` |
+| `generator.py` | Agents that draft/refine evaluator configs from natural language |
+| `datagen.py` | Synthetic dataset row generation with suggested labels |
+| `seeding.py` | Pure shape derivation between an evaluator and a dataset |
+| `runner.py` / `experiment.py` | The two run-execution engines |
+| `metrics.py` | Agreement metrics, pure functions |
+| `schema_migration.py` | Dataset shape/label migration on edit |
+| `spec.py` | Canonical translation to/from `pydantic_evals`/eval-package formats |
+| `export.py` | Render an evaluator or dataset to a standalone runnable script |
+| `tools.py` / `capabilities.py` | What an evaluator agent can be given |
+| `tracing.py` | The only module that talks to Logfire's instrumentation |
+| `logfire_pull.py` / `logfire_io.py` | Logfire traces and hosted datasets in/out |
+| `local_cli/` | Local CLI agent adapters and the bridge model |
+| `api/` | FastAPI app, routers, dependencies |
+| `cli/` | The `valcore` click CLI, a thin shell over the same library |
+
+Frontend, `web/src/`: `pages/` (one per top-level UI surface — Overview,
+Evaluators, Datasets, Runs, Compare, Settings), `components/` (the pieces
+those pages are built from), `api/` (the typed client every request goes
+through).

valcore-0.2.0/CONTRIBUTING.md

@@ -0,0 +1,120 @@
+# Contributing to valcore
+
+valcore is a local-first tool for developing, improving, and running agentic
+evaluations. This document covers the practical mechanics of contributing code.
+For the project's architecture and the philosophy behind it, see
+[AGENTS.md](AGENTS.md); for user-facing behavior, see [README.md](README.md).
+
+## Setup
+
+The backend is a `uv`-managed Python package; the web UI is a separate Vite +
+React SPA.
+
+```bash
+uv sync --all-extras      # installs the dev dependency group plus every extra (e.g. logfire)
+cd web && npm install
+```
+
+`uv sync` installs the `dev` dependency group by default (pytest, ruff,
+pre-commit, …); `--all-extras` additionally pulls in optional extras like
+`logfire`, which some tests exercise.
+
+Install the git hooks once so formatting and linting run automatically:
+
+```bash
+uv run pre-commit install
+```
+
+## Running the app locally
+
+```bash
+uv run valcore serve --no-browser   # API + built-in web UI on :8000
+```
+
+To iterate on the SPA with hot reload instead, run the API and the Vite dev
+server side by side — `web/vite.config.ts` proxies `/api` to `127.0.0.1:8000`:
+
+```bash
+uv run valcore serve --no-browser   # terminal 1
+cd web && npm run dev               # terminal 2, :5173
+```
+
+## Tests
+
+```bash
+uv run pytest              # backend: routes, store, CLI, generation, Logfire I/O, ...
+cd web && npx vitest run   # frontend: components and pages
+cd web && npx tsc --noEmit # frontend: typecheck (also runs as part of `npm run build`)
+```
+
+The backend suite has one test file per source module (`store.py` →
+`tests/test_store.py`, `src/valcore/api/routes/datasets.py` →
+`tests/test_api_datasets.py`, and so on) — add new tests alongside the module
+they cover rather than starting a new grouping. Tests are written test-first:
+add the failing test for the behavior you want, watch it fail for the right
+reason, then write the minimal code that passes it. `tests/test_local_cli_live.py`
+is the one file that talks to real local CLIs (`claude`, `codex`,
+`cursor-agent`) instead of stubbing them — it is skipped unless the relevant
+CLI is actually installed and authenticated, so it will not fail your PR for
+lacking one.
+
+Frontend components ship alongside a co-located `*.test.tsx`, exercised with
+Testing Library — assert on rendered behavior (labels, roles, values a user
+would see), not on implementation details.
+
+## Formatting and linting
+
+Ruff (configured for a 100-column line length) handles both linting and
+formatting; the pre-commit hooks run `ruff check --fix` and `ruff format`
+automatically on commit. `pytest` runs on push, not on every commit — it is
+fast but not instant, and a slow `git commit` is how people learn to pass
+`--no-verify`, which this project asks you not to do.
+
+If you ever need to run everything CI runs, in order:
+
+```bash
+uv run pre-commit run --all-files
+uv run pytest -q
+cd web && npm run build && npx vitest run
+```
+
+## Code style
+
+- Module and function docstrings explain *why*, not *what* — a reader can see
+  what a function does from its name and body; the docstring earns its place
+  by stating a non-obvious constraint, invariant, or the reason a design
+  choice was made a particular way. Most modules in this codebase open with a
+  short docstring doing exactly that; read a few nearby before adding one of
+  your own.
+- Inline comments are rare and reserved for the same reason: a hidden gotcha,
+  not a restatement of the next line.
+- Don't add error handling, fallbacks, or abstraction for cases that can't
+  happen here — trust the internal contracts the store, models, and API
+  layers already establish, and validate only at real boundaries (user input,
+  an external API response).
+- If you rename or add a CLI command, run `python scripts/sync_command_table.py`
+  to update the command table in README.md from the canonical one in
+  `src/valcore/skills/use-valcore/reference.md` (that file is the source of
+  truth; README carries a copy for readers who never install the skill).
+  `tests/test_docs.py` runs the script with `--check` and fails if the two
+  have drifted.
+
+## Branches, commits, and PRs
+
+Branch names are short and descriptive; a `feat/`, `fix/`, or `docs/` prefix
+is common in this repo's history but not enforced. Commit messages favor
+explaining *why* a change was made over restating *what* changed line by
+line.
+
+Every PR runs three CI jobs (`.github/workflows/test.yml`): `pre-commit
+run --all-files` (lint), the backend test matrix across Python 3.11–3.14, and
+the web build + `vitest run`. All three must pass before merge. There is no
+separate release step to run yourself — pushing a `v*` tag on `main` triggers
+`.github/workflows/release.yml`, which builds the SPA, builds the wheel and
+sdist, and publishes to PyPI (and, for a non-prerelease version, updates the
+Homebrew tap).
+
+## License
+
+By contributing, you agree your contribution is licensed under this project's
+[Apache 2.0 license](LICENSE).