pymelvil 0.3.0__tar.gz

pymelvil-0.3.0/.github/workflows/release.yml

@@ -0,0 +1,40 @@
+name: release
+
+on:
+  release:
+    types: [published]
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - name: Build sdist and wheel
+        run: |
+          pip install build
+          python -m build
+      - name: Check metadata
+        run: |
+          pip install twine
+          twine check dist/*
+      - uses: actions/upload-artifact@v4
+        with:
+          name: dist
+          path: dist/
+
+  publish:
+    needs: build
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write
+    steps:
+      - uses: actions/download-artifact@v4
+        with:
+          name: dist
+          path: dist/
+      - name: Publish to PyPI (trusted publishing)
+        uses: pypa/gh-action-pypi-publish@release/v1

pymelvil-0.3.0/.github/workflows/test.yml

@@ -0,0 +1,25 @@
+name: test
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.10", "3.11", "3.12", "3.13"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install
+        run: pip install -e '.[dev]'
+      - name: Lint
+        run: ruff check .
+      - name: Test (offline, fake LMs, no API keys)
+        run: pytest

pymelvil-0.3.0/.gitignore

@@ -0,0 +1,18 @@
+__pycache__/
+*.pyc
+*.egg-info/
+dist/
+build/
+.venv/
+.dspy_cache/
+runs/
+.pytest_cache/
+.ruff_cache/
+.ipynb_checkpoints/
+.env
+runs_iter/
+runs_confirm/
+benchmarks/runs/
+benchmarks/*.log
+_export/
+_snapshot.tar.gz

pymelvil-0.3.0/CODE_OF_CONDUCT.md

@@ -0,0 +1,4 @@
+# Code of Conduct
+
+This project follows the [Contributor Covenant v2.1](https://www.contributor-covenant.org/version/2/1/code_of_conduct/).
+Instances of unacceptable behavior may be reported to harish.siravuri@gmail.com.

pymelvil-0.3.0/CONTRIBUTING.md

@@ -0,0 +1,36 @@
+# Contributing to melvil
+
+Thanks for considering a contribution!
+
+## Setup
+
+```bash
+python -m venv .venv && source .venv/bin/activate
+pip install -e '.[dev]'
+pytest            # must stay green with NO API keys (fake-LM offline suite)
+ruff check .
+```
+
+## Ground rules
+
+- **No API keys in tests.** Anything touching an LLM in tests goes through
+  `melvil.testing` fakes. If your feature can't be exercised offline,
+  extend the fakes first.
+- **The artifact is the contract.** Changes to `PromptArtifact` fields bump
+  `schema_version` and keep `load()` backwards-compatible.
+- **Budget honesty.** Any code path that spends metric calls must account for
+  them in the artifact's `budget` block.
+- Type hints on the public API; `ruff check .` clean; match the surrounding
+  style.
+
+## Where things live
+
+See the module map in [docs/api.md](docs/api.md) and the architecture notes in
+module docstrings — `adapter.py` (gepa integration), `features/` (the
+classification layer), `proposer.py` (reflection prompting).
+
+## Benchmarks
+
+`benchmarks/` is a fixed matrix (datasets × arms × seeds) driven by the
+library. If you claim a quality improvement, add or rerun the relevant arm and
+include the numbers (and their cost) in the PR.

pymelvil-0.3.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Harish Varma Siravuri
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

pymelvil-0.3.0/PKG-INFO

@@ -0,0 +1,161 @@
+Metadata-Version: 2.4
+Name: pymelvil
+Version: 0.3.0
+Summary: Optimized, versioned classifier prompts from labeled examples — GEPA prompt evolution with a classification-specific layer.
+Project-URL: Homepage, https://github.com/harishsiravuri/melvil
+Project-URL: Issues, https://github.com/harishsiravuri/melvil/issues
+Author-email: Harish Varma Siravuri <harish.siravuri@gmail.com>
+License: MIT License
+        
+        Copyright (c) 2026 Harish Varma Siravuri
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+License-File: LICENSE
+Keywords: classification,dspy,gepa,llm,prompt-optimization
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Requires-Dist: dspy>=3.0
+Requires-Dist: gepa>=0.0.20
+Requires-Dist: pyyaml>=6.0
+Provides-Extra: dev
+Requires-Dist: datasets>=3.0; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Requires-Dist: ruff>=0.6; extra == 'dev'
+Provides-Extra: hf
+Requires-Dist: datasets>=3.0; extra == 'hf'
+Description-Content-Type: text/markdown
+
+# melvil
+
+**Labeled examples + a label taxonomy in → an optimized, versioned classifier prompt out.**
+
+melvil wraps the [GEPA](https://github.com/gepa-ai/gepa) reflective prompt-evolution
+engine with a classification-specific layer that generic prompt optimizers lack:
+confusion-driven reflection, a per-label prompt codebook, and hard-example mining.
+It is a pure Python library — the API is the product.
+
+*Named for [Melvil Dewey](https://en.wikipedia.org/wiki/Melvil_Dewey), who gave
+libraries a system for putting things in the right category.*
+
+```python
+import melvil as mv
+
+examples = mv.load_csv("tickets.csv")                  # text,label columns
+train, dev = mv.train_dev_split(examples, dev_size=100, seed=0)
+spec = mv.TaskSpec.from_examples("ticket-intents", examples)
+cfg = mv.Config(task_model="openai/gpt-4.1-mini",
+                reflection_model="openai/gpt-4.1", budget="light")
+artifact = mv.optimize(spec, train, dev, cfg)
+print(artifact.render())                               # deployable prompt string
+artifact.save("ticket_intents.v1.json")
+```
+
+## Install
+
+```bash
+pip install pymelvil             # the PyPI distribution is `pymelvil`; you `import melvil`
+pip install 'pymelvil[hf]'       # + HuggingFace dataset loaders
+```
+
+From a checkout:
+
+```bash
+pip install -e '.[hf,dev]'       # + test/lint tooling
+```
+
+Model names are [LiteLLM](https://docs.litellm.ai) ids (`openai/...`,
+`anthropic/...`, `openrouter/...`); set the matching API key env var
+(`OPENAI_API_KEY`, `OPENROUTER_API_KEY`, ...).
+
+## What the benchmarks say (read this before choosing features)
+
+We benchmark honestly, including against ourselves — full protocol and numbers in
+[benchmarks/RESULTS.md](benchmarks/RESULTS.md). The pre-registered confirmation pass
+(8 public datasets, fresh seeds, light budget, gpt-4.1-mini) found:
+
+- **Vanilla GEPA over melvil's rendered prompt is the strongest configuration**
+  (mean test accuracy 0.781 vs 0.755 for the full classification layer and 0.762 for
+  MIPROv2; seed prompt 0.703). At light budgets, prefer `features=mv.Features.none()`.
+- The classification layer's per-component updates trade whole-prompt coverage for
+  structure; at ~6–10 accepted proposals per run that trade loses, especially on hard
+  small-taxonomy tasks. Whether it wins at medium/heavy budgets is an open question.
+- Hard-example mining, behind its (strict, quarantined) accept gate, kept exemplars in
+  0/24 confirmation runs — treat it as a safety-gated no-op at light budgets.
+
+## What the classification layer does
+
+Each feature is independently toggleable via `mv.Features` (**all off by default** — the default is the benchmark-strongest vanilla-GEPA configuration; enable the layer with `mv.Features.all()` or per-flag;
+`Features.none()` is an explicit alias for the default):
+
+1. **Per-label codebook** — the prompt is not a free-text blob but named
+   components: a task instruction, one definition per label, and boundary
+   rules. GEPA evolves them per-component.
+2. **Confusion-driven reflection** — every full dev evaluation updates a
+   confusion matrix; reflection rounds are pointed at the label components on
+   both sides of the currently worst confused boundary, and the reflection LM
+   is shown the top confused pairs with concrete misclassified examples.
+3. **Hard-example mining** — dev examples that stay misclassified across the
+   run become candidate few-shot exemplars, selected to cover the top confused
+   boundaries, and kept only if they don't hurt dev accuracy. (Exemplars come
+   from dev, so dev scores of exemplar-augmented artifacts are mildly
+   optimistic — judge them on test.)
+
+The rendered prompt always ends with a fixed, non-evolvable output-format
+contract, so the optimizer can never break parseability.
+
+## Everything else you get
+
+- **`PromptArtifact`** — versioned JSON: components, models, budget and cost
+  actually spent, dev scores + confusion matrix, the dev-score-vs-budget curve,
+  config hash, lineage (`parent_id`). `artifact.diff(other)` gives a
+  per-component diff with score deltas.
+- **`evaluate(artifact, data, model=...)`** — accuracy, macro-F1, per-label
+  P/R/F1, confusion, cost; pass a different `model` for a transfer evaluation.
+  `report(...)` renders it as markdown.
+- **Cost estimation before spending** — `estimate_optimize_cost(spec, train,
+  dev, cfg)` is a dry-run upper bound; measured spend comes from the LM call
+  history and lands in the artifact.
+- **Run directories & resume** — every run writes `runs/<task>/<hash>-s<seed>/`
+  (config, engine state, reflection traces, artifact);
+  `optimize(..., resume=True)` continues an interrupted run.
+- **Progress** — pass `on_round=lambda info: ...` for live dev score / spend
+  after every full dev evaluation, or just read the default logging.
+- **Offline testing** — `melvil.testing` ships fake LMs with a real
+  optimization gradient; the whole test suite runs with no API keys.
+
+## Worked example & docs
+
+- [examples/quickstart.py](examples/quickstart.py) — the snippet above, runnable end-to-end on AG News.
+- [examples/agnews_demo.ipynb](examples/agnews_demo.ipynb) — notebook walkthrough with a live progress callback.
+- [docs/api.md](docs/api.md) — public API reference.
+- [benchmarks/](benchmarks/) — the honest-benchmark harness (datasets × arms × seeds).
+
+## Development
+
+```bash
+pip install -e '.[dev]'
+pytest          # green with no API keys — fake-LM offline suite
+ruff check .
+```
+
+License: MIT.

pymelvil-0.3.0/README.md

@@ -0,0 +1,114 @@
+# melvil
+
+**Labeled examples + a label taxonomy in → an optimized, versioned classifier prompt out.**
+
+melvil wraps the [GEPA](https://github.com/gepa-ai/gepa) reflective prompt-evolution
+engine with a classification-specific layer that generic prompt optimizers lack:
+confusion-driven reflection, a per-label prompt codebook, and hard-example mining.
+It is a pure Python library — the API is the product.
+
+*Named for [Melvil Dewey](https://en.wikipedia.org/wiki/Melvil_Dewey), who gave
+libraries a system for putting things in the right category.*
+
+```python
+import melvil as mv
+
+examples = mv.load_csv("tickets.csv")                  # text,label columns
+train, dev = mv.train_dev_split(examples, dev_size=100, seed=0)
+spec = mv.TaskSpec.from_examples("ticket-intents", examples)
+cfg = mv.Config(task_model="openai/gpt-4.1-mini",
+                reflection_model="openai/gpt-4.1", budget="light")
+artifact = mv.optimize(spec, train, dev, cfg)
+print(artifact.render())                               # deployable prompt string
+artifact.save("ticket_intents.v1.json")
+```
+
+## Install
+
+```bash
+pip install pymelvil             # the PyPI distribution is `pymelvil`; you `import melvil`
+pip install 'pymelvil[hf]'       # + HuggingFace dataset loaders
+```
+
+From a checkout:
+
+```bash
+pip install -e '.[hf,dev]'       # + test/lint tooling
+```
+
+Model names are [LiteLLM](https://docs.litellm.ai) ids (`openai/...`,
+`anthropic/...`, `openrouter/...`); set the matching API key env var
+(`OPENAI_API_KEY`, `OPENROUTER_API_KEY`, ...).
+
+## What the benchmarks say (read this before choosing features)
+
+We benchmark honestly, including against ourselves — full protocol and numbers in
+[benchmarks/RESULTS.md](benchmarks/RESULTS.md). The pre-registered confirmation pass
+(8 public datasets, fresh seeds, light budget, gpt-4.1-mini) found:
+
+- **Vanilla GEPA over melvil's rendered prompt is the strongest configuration**
+  (mean test accuracy 0.781 vs 0.755 for the full classification layer and 0.762 for
+  MIPROv2; seed prompt 0.703). At light budgets, prefer `features=mv.Features.none()`.
+- The classification layer's per-component updates trade whole-prompt coverage for
+  structure; at ~6–10 accepted proposals per run that trade loses, especially on hard
+  small-taxonomy tasks. Whether it wins at medium/heavy budgets is an open question.
+- Hard-example mining, behind its (strict, quarantined) accept gate, kept exemplars in
+  0/24 confirmation runs — treat it as a safety-gated no-op at light budgets.
+
+## What the classification layer does
+
+Each feature is independently toggleable via `mv.Features` (**all off by default** — the default is the benchmark-strongest vanilla-GEPA configuration; enable the layer with `mv.Features.all()` or per-flag;
+`Features.none()` is an explicit alias for the default):
+
+1. **Per-label codebook** — the prompt is not a free-text blob but named
+   components: a task instruction, one definition per label, and boundary
+   rules. GEPA evolves them per-component.
+2. **Confusion-driven reflection** — every full dev evaluation updates a
+   confusion matrix; reflection rounds are pointed at the label components on
+   both sides of the currently worst confused boundary, and the reflection LM
+   is shown the top confused pairs with concrete misclassified examples.
+3. **Hard-example mining** — dev examples that stay misclassified across the
+   run become candidate few-shot exemplars, selected to cover the top confused
+   boundaries, and kept only if they don't hurt dev accuracy. (Exemplars come
+   from dev, so dev scores of exemplar-augmented artifacts are mildly
+   optimistic — judge them on test.)
+
+The rendered prompt always ends with a fixed, non-evolvable output-format
+contract, so the optimizer can never break parseability.
+
+## Everything else you get
+
+- **`PromptArtifact`** — versioned JSON: components, models, budget and cost
+  actually spent, dev scores + confusion matrix, the dev-score-vs-budget curve,
+  config hash, lineage (`parent_id`). `artifact.diff(other)` gives a
+  per-component diff with score deltas.
+- **`evaluate(artifact, data, model=...)`** — accuracy, macro-F1, per-label
+  P/R/F1, confusion, cost; pass a different `model` for a transfer evaluation.
+  `report(...)` renders it as markdown.
+- **Cost estimation before spending** — `estimate_optimize_cost(spec, train,
+  dev, cfg)` is a dry-run upper bound; measured spend comes from the LM call
+  history and lands in the artifact.
+- **Run directories & resume** — every run writes `runs/<task>/<hash>-s<seed>/`
+  (config, engine state, reflection traces, artifact);
+  `optimize(..., resume=True)` continues an interrupted run.
+- **Progress** — pass `on_round=lambda info: ...` for live dev score / spend
+  after every full dev evaluation, or just read the default logging.
+- **Offline testing** — `melvil.testing` ships fake LMs with a real
+  optimization gradient; the whole test suite runs with no API keys.
+
+## Worked example & docs
+
+- [examples/quickstart.py](examples/quickstart.py) — the snippet above, runnable end-to-end on AG News.
+- [examples/agnews_demo.ipynb](examples/agnews_demo.ipynb) — notebook walkthrough with a live progress callback.
+- [docs/api.md](docs/api.md) — public API reference.
+- [benchmarks/](benchmarks/) — the honest-benchmark harness (datasets × arms × seeds).
+
+## Development
+
+```bash
+pip install -e '.[dev]'
+pytest          # green with no API keys — fake-LM offline suite
+ruff check .
+```
+
+License: MIT.

pymelvil-0.3.0/SECURITY.md

@@ -0,0 +1,8 @@
+# Security Policy
+
+melvil makes LLM API calls with keys read from environment variables; it never
+stores or transmits keys elsewhere. Run directories and artifacts contain your
+dataset text and model outputs — treat them with the same care as the data itself.
+
+To report a vulnerability, email harish.siravuri@gmail.com rather than opening a
+public issue. You should receive a response within a week.