driftless 0.1.0__tar.gz

driftless-0.1.0/.gitignore

@@ -0,0 +1,22 @@
+# Python
+__pycache__/
+*.py[cod]
+*.egg-info/
+.eggs/
+build/
+dist/
+.venv/
+venv/
+.env
+
+# Tooling caches
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+
+# driftless run artifacts
+.driftless/
+
+# OS / editor
+.DS_Store
+.scratch/

driftless-0.1.0/IMPLEMENTATION_PLAN.md

@@ -0,0 +1,97 @@
+# Implementation Plan — Dependabot for LLM Models (`driftless`)
+
+A GitHub-native tool that detects risky/deprecated/expensive LLM model
+dependencies, evaluates replacement models through the **real application
+path**, repairs prompts/configs where allowed, validates against thresholds,
+and opens migration PRs with evidence.
+
+## Build philosophy
+
+- **CLI-first.** The CLI is the engine. The GitHub Action and App just invoke
+  it. Everything is testable locally before any GitHub integration exists.
+- **The workflow contract is the spine.** A single typed schema
+  (`driftless.yml`) drives `scan`, `compare`, `migrate`, `validate`, and
+  `report`. Build it first and rigorously.
+- **The customer owns the workflow; we orchestrate.** We shell out to *their*
+  eval command with the model overridden via env var. We never reimplement
+  their preprocessing/parsing/postprocessing.
+- **Failure is a first-class output.** Every command can emit a
+  `pass` / `partial` / `blocked` result, not just success.
+
+## Stack
+
+- **Language:** Python (3.10+)
+- **CLI framework:** Typer
+- **Validation:** Pydantic v2
+- **Config:** YAML (`driftless.yml`)
+- **Tests:** pytest
+- **Distribution:** pip / pipx; GitHub Action wraps the same CLI.
+
+## Milestones
+
+### Milestone 1 — Contract + harness (foundation)
+- Project scaffold, CLI entrypoint, config loader.
+- Workflow contract schema (`workflows`, `model`, `files.editable/readonly`,
+  `eval`, `thresholds`, `migration`) with strict validation + good errors.
+- Runnable harness: execute the user's `command` with the model injected via
+  env var; read `input_path`, capture `output_path`; handle exit codes/timeouts.
+- `driftless init` (scaffold a contract) and `driftless validate`
+  (contract parses + harness runs with current model).
+- **Done when:** we can run the current model through the real command and
+  capture production-shaped outputs.
+
+### Milestone 2 — Evaluation + comparison
+- Output loaders + schema validation (JSON schema), enum/field-level checks.
+- Metrics: accuracy, precision, recall, F1, schema-error rate, refusal rate;
+  cost + latency capture per run.
+- `driftless compare --workflow X --to <model>`: baseline vs naive target
+  scorecard.
+- **Done when:** `compare` reproduces the "Target, Original Files" column.
+
+### Milestone 3 — Migration loop (port validated work)
+- Failure clustering over baseline-vs-target diff.
+- Candidate patch generation scoped strictly to `files.editable`.
+- Tuning/holdout split, iterate <= `max_iterations`, `select_best` under
+  constraints; holdout validation as a hard gate.
+- `driftless migrate --workflow X --to <model>` -> migrated files + result
+  object (`pass` / `partial` / `blocked`).
+- **Done when:** on validated workflows, `migrate` recovers performance and
+  only edits allowed files.
+
+### Milestone 4 — Reporting (incl. failure path)
+- Markdown migration report: before/after metrics, changes made, remaining
+  risks, fixed/remaining examples, cost/latency.
+- Distinct, useful artifacts for partial/blocked outcomes (remaining clusters,
+  recommended fallback model).
+- `driftless report`.
+- **Done when:** a partial migration still produces something a partner acts on.
+
+### Milestone 5 — Static scanner
+- Detect probable LLM usage (SDK imports, model-ID literals, env-var config).
+- Provider lifecycle data (deprecation/retirement) as static JSON first,
+  structured to later become a hosted DB.
+- `driftless scan` + `configure` helper to scaffold a contract from a
+  detected workflow.
+- **Done when:** `scan` surfaces model usage + deprecation risk with no
+  contract required.
+
+### Milestone 6 — GitHub surface
+- GitHub Action wrapping `scan`/`compare`/`migrate`.
+- PR generation: branch, commit migrated files, evidence-rich PR body. Open
+  issue/draft PR on partial/blocked.
+- **Done when:** a real migration opens a real PR with the evidence table.
+
+Milestones 1–4 yield a locally usable, dogfoodable tool. 5–6 make it the
+Dependabot-style experience. Paid App/dashboard/billing are earned scope after.
+
+## CLI surface (target)
+
+```
+driftless init
+driftless scan
+driftless configure <workflow>
+driftless compare  --workflow <w> --to <model>
+driftless migrate  --workflow <w> --to <model>
+driftless validate --workflow <w>
+driftless report
+```

driftless-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 driftless contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

driftless-0.1.0/PKG-INFO

@@ -0,0 +1,137 @@
+Metadata-Version: 2.4
+Name: driftless
+Version: 0.1.0
+Summary: Keep prompts in sync when model or eval data changes — Poetry-style lock regeneration, Dependabot-style PRs.
+Project-URL: Homepage, https://github.com/driftless/driftless
+Project-URL: Repository, https://github.com/driftless/driftless
+Project-URL: Documentation, https://github.com/driftless/driftless/blob/main/docs/repair-and-generators.md
+Project-URL: Issues, https://github.com/driftless/driftless/issues
+Author: driftless contributors
+License: MIT
+License-File: LICENSE
+Keywords: anthropic,ci,dependabot,evaluation,llm,migration,openai,prompts
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Quality Assurance
+Requires-Python: >=3.10
+Requires-Dist: jsonschema>=4.21
+Requires-Dist: pydantic>=2.6
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: rich>=13.7
+Requires-Dist: typer>=0.12
+Provides-Extra: dev
+Requires-Dist: build>=1.2; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Provides-Extra: llm
+Requires-Dist: anthropic>=0.30; extra == 'llm'
+Requires-Dist: openai>=1.0; extra == 'llm'
+Description-Content-Type: text/markdown
+
+# driftless
+
+**Poetry-style lock regeneration for prompts — delivered Dependabot-style.**
+
+A prompt is pinned to a **model** and an **eval dataset** (like `pyproject.toml`
+declares deps and `poetry.lock` pins what works). When either moves, the prompt
+goes stale. driftless re-derives it through your real eval, validates on holdout,
+and opens a PR with evidence.
+
+> Also described as *Dependabot for LLM models* — same automation shape, different
+> core insight: prompts are lockfiles, not just config files.
+
+> Status: early development. See [`IMPLEMENTATION_PLAN.md`](./IMPLEMENTATION_PLAN.md).
+
+## Install (dev)
+
+```bash
+python -m venv .venv
+source .venv/bin/activate
+pip install -e ".[dev]"
+```
+
+## Quickstart
+
+```bash
+driftless init            # scaffold a driftless.yml
+driftless validate -w support_classifier   # contract parses + harness runs
+```
+
+## How it works
+
+You describe your model-dependent workflow once in `driftless.yml`: how to
+run it, how to override the model, which files may be edited, and what quality
+thresholds must hold. `driftless` orchestrates *your* workflow under
+different models, compares results, repairs allowed files, validates on
+holdout, and opens a PR with the evidence.
+
+The customer owns the workflow. The tool orchestrates it.
+
+Not a classifier? Choose a grading mode that fits the task — the same loop then
+optimizes against it, with your team owning the definition of "good":
+
+- **`eval.score_field` / `eval.pass_field`** — your command emits a numeric score
+  or a pass/fail per record (works for any task: summarization, codegen, agents).
+- **`eval.fields`** — structured extraction, scored per field with
+  precision/recall/F1 against the gold record.
+- **`eval.judge`** — an LLM judge grades each free-form output against a rubric
+  (with an optional human-scored calibration set for a judge-agreement check).
+
+## CLI
+
+| Command | Purpose |
+|---|---|
+| `init` | Scaffold a `driftless.yml`. |
+| `init-policy` | Scaffold a `.driftless/policy.yml` (when to migrate). |
+| `scan` | Find probable LLM usage and at-risk models. |
+| `plan` | Discover at-risk workflows and apply the migration policy (CI triage). |
+| `plan --act` | Migrate + open a PR/issue for every actionable trigger (close the loop). |
+| `configure <workflow>` | Turn a detected workflow into a migration-ready contract. |
+| `calibrate -w <w>` | Measure the baseline and suggest starting thresholds. |
+| `compare -w <w> --to <model>` | Baseline vs target scorecard. |
+| `migrate -w <w> --to <model>` | Repair + validate + produce migrated files. |
+| `refine -w <w>` | Re-optimize the prompt for a changed eval dataset (model pinned). |
+| `poll [--act]` | Detect external eval-dataset changes and refine on a meaningful change. |
+| `validate -w <w>` | Check the contract parses and the harness runs. |
+| `report` | Render the latest migration report. |
+| `view` | Open the optimization run viewer (charts + attempt log). |
+| `open-pr -w <w>` | Open a PR (or issue) from the latest migration result. |
+
+## Configuring *when* to migrate
+
+`plan` reads an optional `.driftless/policy.yml` — the "dependabot.yml" layer.
+Scaffold it with `driftless init-policy`; every field matches a default, so an
+empty file behaves like no file. It controls which triggers are enabled
+(`deprecation` is on and forced; `cost`/`quality`/`new_model` are opportunistic),
+the thresholds a candidate must clear (`min_savings_pct`, `min_gain`), a
+`cooldown_days` to skip freshly-released models, candidate `allow`/`deny` globs,
+and an `ignore` list to snooze specific models or moves. The engine still decides
+whether a candidate actually passes *your* eval — policy only decides whether to
+propose it.
+
+## GitHub-native usage
+
+A composite GitHub Action (`action.yml`) wraps the CLI so scans and migrations
+can run in CI. See `.github/workflows/` for a scheduled deprecation scan and a
+manually-triggered migration that opens a PR (or an issue when blocked).
+
+```yaml
+- uses: driftless/action@v1
+  with:
+    command: scan
+```
+
+## Documentation
+
+- [Project overview](./docs/PROJECT_OVERVIEW.md) — what's been built so far:
+  architecture, components, the migration loop, and the testbed repo.
+- [Next steps / roadmap](./docs/NEXT_STEPS.md) — prioritized backlog and where
+  each item lands in the code.
+- [Repair prompts & custom generators](./docs/repair-and-generators.md) — customize
+  the LLM repair prompt or plug in your own patch generator.
+- [Implementation plan](./IMPLEMENTATION_PLAN.md) — milestones and architecture.

driftless-0.1.0/README.md

@@ -0,0 +1,102 @@
+# driftless
+
+**Poetry-style lock regeneration for prompts — delivered Dependabot-style.**
+
+A prompt is pinned to a **model** and an **eval dataset** (like `pyproject.toml`
+declares deps and `poetry.lock` pins what works). When either moves, the prompt
+goes stale. driftless re-derives it through your real eval, validates on holdout,
+and opens a PR with evidence.
+
+> Also described as *Dependabot for LLM models* — same automation shape, different
+> core insight: prompts are lockfiles, not just config files.
+
+> Status: early development. See [`IMPLEMENTATION_PLAN.md`](./IMPLEMENTATION_PLAN.md).
+
+## Install (dev)
+
+```bash
+python -m venv .venv
+source .venv/bin/activate
+pip install -e ".[dev]"
+```
+
+## Quickstart
+
+```bash
+driftless init            # scaffold a driftless.yml
+driftless validate -w support_classifier   # contract parses + harness runs
+```
+
+## How it works
+
+You describe your model-dependent workflow once in `driftless.yml`: how to
+run it, how to override the model, which files may be edited, and what quality
+thresholds must hold. `driftless` orchestrates *your* workflow under
+different models, compares results, repairs allowed files, validates on
+holdout, and opens a PR with the evidence.
+
+The customer owns the workflow. The tool orchestrates it.
+
+Not a classifier? Choose a grading mode that fits the task — the same loop then
+optimizes against it, with your team owning the definition of "good":
+
+- **`eval.score_field` / `eval.pass_field`** — your command emits a numeric score
+  or a pass/fail per record (works for any task: summarization, codegen, agents).
+- **`eval.fields`** — structured extraction, scored per field with
+  precision/recall/F1 against the gold record.
+- **`eval.judge`** — an LLM judge grades each free-form output against a rubric
+  (with an optional human-scored calibration set for a judge-agreement check).
+
+## CLI
+
+| Command | Purpose |
+|---|---|
+| `init` | Scaffold a `driftless.yml`. |
+| `init-policy` | Scaffold a `.driftless/policy.yml` (when to migrate). |
+| `scan` | Find probable LLM usage and at-risk models. |
+| `plan` | Discover at-risk workflows and apply the migration policy (CI triage). |
+| `plan --act` | Migrate + open a PR/issue for every actionable trigger (close the loop). |
+| `configure <workflow>` | Turn a detected workflow into a migration-ready contract. |
+| `calibrate -w <w>` | Measure the baseline and suggest starting thresholds. |
+| `compare -w <w> --to <model>` | Baseline vs target scorecard. |
+| `migrate -w <w> --to <model>` | Repair + validate + produce migrated files. |
+| `refine -w <w>` | Re-optimize the prompt for a changed eval dataset (model pinned). |
+| `poll [--act]` | Detect external eval-dataset changes and refine on a meaningful change. |
+| `validate -w <w>` | Check the contract parses and the harness runs. |
+| `report` | Render the latest migration report. |
+| `view` | Open the optimization run viewer (charts + attempt log). |
+| `open-pr -w <w>` | Open a PR (or issue) from the latest migration result. |
+
+## Configuring *when* to migrate
+
+`plan` reads an optional `.driftless/policy.yml` — the "dependabot.yml" layer.
+Scaffold it with `driftless init-policy`; every field matches a default, so an
+empty file behaves like no file. It controls which triggers are enabled
+(`deprecation` is on and forced; `cost`/`quality`/`new_model` are opportunistic),
+the thresholds a candidate must clear (`min_savings_pct`, `min_gain`), a
+`cooldown_days` to skip freshly-released models, candidate `allow`/`deny` globs,
+and an `ignore` list to snooze specific models or moves. The engine still decides
+whether a candidate actually passes *your* eval — policy only decides whether to
+propose it.
+
+## GitHub-native usage
+
+A composite GitHub Action (`action.yml`) wraps the CLI so scans and migrations
+can run in CI. See `.github/workflows/` for a scheduled deprecation scan and a
+manually-triggered migration that opens a PR (or an issue when blocked).
+
+```yaml
+- uses: driftless/action@v1
+  with:
+    command: scan
+```
+
+## Documentation
+
+- [Project overview](./docs/PROJECT_OVERVIEW.md) — what's been built so far:
+  architecture, components, the migration loop, and the testbed repo.
+- [Next steps / roadmap](./docs/NEXT_STEPS.md) — prioritized backlog and where
+  each item lands in the code.
+- [Repair prompts & custom generators](./docs/repair-and-generators.md) — customize
+  the LLM repair prompt or plug in your own patch generator.
+- [Implementation plan](./IMPLEMENTATION_PLAN.md) — milestones and architecture.