modelpin 0.4.2__tar.gz → 0.5.2__tar.gz

{modelpin-0.4.2 → modelpin-0.5.2}/CHANGELOG.md

@@ -8,6 +8,99 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 Nothing yet.
 
+## [0.5.2] - 2026-11-05
+
+The first release on PyPI since 0.5.0: it also carries everything listed under 0.5.1.
+
+### Fixed
+
+- **`modelpin watch` no longer reports as retired four Gemini models that Google's pages still
+  list as available.** The registry carried Vertex AI's 20 October 2026 date for
+  `gemini-2.5-pro`, `gemini-2.5-flash` and `gemini-2.5-flash-lite`, and, for
+  `gemini-2.5-flash-image`, the 2 October 2026 date it read from the Gemini API on 17 September.
+  Once those dates passed (for `gemini-2.5-flash-image`, from the day 0.5.0 shipped), `watch`
+  called all four retired. Read again on 5 November 2026:
+  - `gemini-2.5-pro` and `gemini-2.5-flash`: the Gemini API gives 1 December 2026 as the
+    shutdown date, after which requests to them route to `gemini-3.8-flash` and
+    `gemini-3.5-flash-lite` respectively; access is already limited to earlier users. `watch`
+    now names those successors.
+  - `gemini-2.5-flash-lite`: the Gemini API says it "is not deprecated and remains available
+    through the API until further notice". Vertex AI still lists the passed 20 October date, so
+    `watch` reports it deprecated with no shutdown date, and names `gemini-3.8-flash`, the first
+    successor Vertex lists.
+  - `gemini-2.5-flash-image`: both Google pages now give 15 March 2027, successor
+    `gemini-3.1-flash-lite-image`.
+
+  Where two Google surfaces disagree, the registry still carries the earlier date, but no longer
+  once that date has passed while the other surface still lists the model as available.
+
+## [0.5.1] - 2026-10-04
+
+Tagged and released on GitHub, but never published to PyPI: the publish job waited for the
+required `pypi` approval, none was given, and GitHub failed it after 30 days. These changes
+first reach PyPI in 0.5.2.
+
+### Fixed
+
+- **Tool scenarios should now replay on `gpt-5.6-sol`, `gpt-5.6-terra`, `gpt-5.6-luna`,
+  `gpt-6-sol` and `gpt-6-luna`.** According to OpenAI's and Azure OpenAI's documentation, Chat
+  Completions accepts tools from these models only with `reasoning_effort: none` and rejects the
+  request otherwise (for gpt-5.6, even when no effort is sent). So every tool scenario in a
+  check against the successor OpenAI names for its 23 October 2026 retirements would come back
+  "could not replay"; this has not yet been confirmed by a live call. Modelpin now sends `none`
+  when a scenario sends tools to one of them and no effort was chosen, and the reports say so.
+  With no `reasoning_effort` set, no other request changes, apart from the `gpt-6*` change
+  below.
+- `gpt-6*` ids are now treated as reasoning models: they get `max_completion_tokens` and no
+  `temperature`.
+
+### Added
+
+- **`reasoning_effort` for OpenAI reasoning models**, in `modelpin.yaml` or as
+  `--reasoning-effort` on `baseline` and `check`, so a check measures the effort you will ship.
+  It is sent only to reasoning models (gpt-5.x, gpt-6, o-series); a scenario can set its own.
+  `last-report.md` (the pull-request comment) and `migration-report.md` state the effort each
+  side was sent, read from the recorded runs. `modelpin report` does not read the setting yet. Choosing an effort other than `none` for a tool scenario on one of the
+  models above stops that scenario before any call, because Chat Completions would reject it.
+  The Responses API is not supported yet.
+
+- **A migration report for the person who decides on the switch.** Whenever `modelpin check`
+  writes `last-report.md`, it now also writes `.modelpin/migration-report.md`: the same run in
+  plain English, with a one-line bottom line, a one-paragraph verdict, each changed scenario
+  with what it means for your app and a before and after example, what the check did not cover
+  (untested behavior, changes that show up in only some runs, scenarios that could not be
+  measured or compared), and the run's date, both models, runs per scenario, judge calls and
+  tokens. It is a renderer over the verdicts `check` already computes and changes no verdict or
+  exit code.
+
+## [0.5.0] - 2026-10-04
+
+### Added
+
+- **`modelpin watch`: which of your models are retiring, when, and what to check next.** It
+  crosses the models in `modelpin.yaml` (its `models:` and `judge_model`; with `--scan`, the
+  ids in your source; recorded baselines are listed, never alarmed on) with a registry shipped
+  inside the package, prints days remaining, the successor the vendor names, and the exact
+  `modelpin check --from ... --to ...` line, and exits `1` inside a notice window or already
+  retired, `3` for a model the registry does not know (unknown is not a clearance), `0` when
+  all clear, `4` when nothing is declared or the config or registry cannot be read. `--json` for scripts; `--registry <path>` to use a different `data/models.json`.
+  It never touches the network.
+- **The model registry now carries real dates, and every entry carries its source.** 87
+  sourced entries (55 OpenAI, 16 Anthropic, 16 Google), a transcription of the vendor sections
+  named per entry rather than a complete catalogue, each naming the page it was read from and
+  the day it was read; `Model` refuses an entry without both, so an unsourced claim cannot enter
+  from the shipped seed or from a `--registry` file. Vendor aliases are recorded, so `gpt-4`
+  finds the `gpt-4-0613` row and `gpt-4o-mini` its dated snapshot; image and audio models carry
+  a `modality` so no chat check is suggested for them. The old seed's two OpenAI ids, `gpt-5.2`
+  and `gpt-5.5` (its note declared every OpenAI and Google id and every date UNVERIFIED),
+  appeared on none of the vendor lifecycle pages fetched on 2026-09-17 and were removed as
+  entries; the dated snapshots behind those aliases were added from their model pages.
+  `scripts/registry_check.py` validates the JSON and regenerates the Python mirror. The
+  OpenAI entries were re-read on 2026-10-04: the 2026-10-23 shutdowns are unchanged, GPT-5.1
+  and GPT-5.4-Nano are now deprecated with a 2027-04-01 shutdown, the GPT Image rows name
+  their new replacements, and `gpt-5.4-cyber` no longer names a successor, because the page
+  no longer does.
+
 ## [0.4.2] - 2026-10-01
 
 Two changes. Python 3.11 is now supported. And a fix that affects you only if you set
@@ -1474,7 +1567,10 @@ not the exit code.
   opinion-framed Markdown + JSON report.
 - BYO-key throughout, with key-shaped-secret scrubbing on all output.
 
-[Unreleased]: https://github.com/samarthputhraya/modelpin/compare/v0.4.2...HEAD
+[Unreleased]: https://github.com/samarthputhraya/modelpin/compare/v0.5.2...HEAD
+[0.5.2]: https://github.com/samarthputhraya/modelpin/compare/v0.5.1...v0.5.2
+[0.5.1]: https://github.com/samarthputhraya/modelpin/compare/v0.5.0...v0.5.1
+[0.5.0]: https://github.com/samarthputhraya/modelpin/compare/v0.4.2...v0.5.0
 [0.4.2]: https://github.com/samarthputhraya/modelpin/compare/v0.4.1...v0.4.2
 [0.4.1]: https://github.com/samarthputhraya/modelpin/compare/v0.4.0...v0.4.1
 [0.4.0]: https://github.com/samarthputhraya/modelpin/compare/v0.3.1...v0.4.0

{modelpin-0.4.2 → modelpin-0.5.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: modelpin
-Version: 0.4.2
+Version: 0.5.2
 Summary: Dependabot for AI models — catch model-migration regressions before they hit production.
 Author: Samarth Puthraya
 License-Expression: Apache-2.0
@@ -85,8 +85,17 @@ OK 6 scenario(s) unchanged
   Modelpin or any third party other than the model providers you configure (and, in CI, your own
   pull request).
 
-Modelpin does not watch provider release or retirement feeds and does not open pull requests on
-its own: you choose the candidate model, and a scheduled workflow can re-check it on your clock.
+`modelpin watch` crosses the models your config names (and, with `--scan`, the ones in your
+source) with a registry shipped inside each release, and tells you which of the ones **it knows**
+are retiring, when, the successor the vendor names, and the exact check to run next. Every entry
+in that registry carries the vendor page it was read from and the day it was read; a model the
+registry has never heard of is reported as unknown, never as clear. The registry is transcribed
+from OpenAI, Anthropic and Google's own deprecation and model pages, not from the catalogues
+of OpenAI-compatible
+hosts. **Nothing in Modelpin polls provider feeds**: the registry is refreshed when a release
+ships, so it is only as current as the release you installed (`--registry <path>` points at a
+newer file). Modelpin does not open pull requests on its own: you choose the candidate model,
+and a scheduled workflow can re-check it on your clock.
 
 CLI: `modelpin` (alias `mp`). License: Apache-2.0.
 
@@ -114,7 +123,7 @@ Python 3.11 or newer.
 
 ```bash
 pip install "modelpin[providers]"      # or: pipx install "modelpin[providers]"
-modelpin version                        # -> modelpin 0.4.2
+modelpin version                        # -> modelpin 0.5.2
 ```
 
 The `providers` extra installs the OpenAI, Anthropic and Google SDKs. Plain `pip install modelpin`
@@ -153,7 +162,10 @@ You will see one scenario per verdict:
 
 The command exits **1** because regressions were found — that is what fails a CI build. The full
 report is written to `.modelpin/last-report.md`, the same file the GitHub Action posts on a pull
-request. Edit `traces.json` and re-run to watch the verdicts move.
+request. Beside it, `.modelpin/migration-report.md` tells the same result in plain English for
+whoever decides on the switch: a one-paragraph verdict, each changed scenario with a before and
+after example, what the check did not cover, and the run's date, models, runs and calls. Edit
+`traces.json` and re-run to watch the verdicts move.
 
 ## Use it on your app
 
@@ -333,7 +345,7 @@ key, is not recognised.
 
 | Provider (`providers:` / `--provider`) | Credentials | Notes |
 |---|---|---|
-| `openai` | `OPENAI_API_KEY` | Chat Completions, multi-turn tool calls. Reasoning models (`o1`/`o3`/`o4`/`gpt-5*`) get `max_completion_tokens` and no temperature |
+| `openai` | `OPENAI_API_KEY` | Chat Completions, multi-turn tool calls. Reasoning models (`o1`/`o3`/`o4`/`gpt-5*`/`gpt-6*`) get `max_completion_tokens` and no temperature |
 | `anthropic` | `ANTHROPIC_API_KEY`, **or** Claude on Vertex AI: `ANTHROPIC_VERTEX_PROJECT_ID` + `gcloud auth application-default login` (`CLOUD_ML_REGION` defaults to `global`) | Messages API, multi-turn tool calls. Newer Claude models accept only default sampling, so a scenario's `temperature` is not sent to them. Verified offline against the SDK's request shapes; not yet validated by a live run |
 | `google` | `GEMINI_API_KEY` (AI Studio), **or** Vertex AI: `GOOGLE_GENAI_USE_VERTEXAI=true` + `GOOGLE_CLOUD_PROJECT` + application-default login | Multi-turn tool calls. On Vertex, `GOOGLE_CLOUD_LOCATION` defaults to `global`; in the project's testing `gemini-3.x` ids were not served on `us-central1`, so set it only if you need a specific region |
 | `groq`, `openrouter`, `together`, `cerebras` | `GROQ_API_KEY`, `OPENROUTER_API_KEY`, `TOGETHER_API_KEY`, `CEREBRAS_API_KEY` | OpenAI-compatible endpoints. Groq has a free tier |
@@ -344,6 +356,17 @@ vendor (`gpt-*`, `claude-*` and `gemini-*` do; `openai/gpt-oss-120b` on Groq doe
 model that is neither the one you run today nor the candidate. Remove `judge_model:` to compare
 only tool calls, refusals and text checks, with no extra calls.
 
+**Reasoning effort** (OpenAI reasoning models): set `reasoning_effort:` in `modelpin.yaml`, or
+pass `--reasoning-effort` to `baseline` and `check` (both read it), to measure the effort you
+will ship (`none`, `minimal`, `low`, `medium`, `high` or `xhigh`; which values a model accepts
+is OpenAI's call). A scenario can set its own `reasoning_effort:`. It is sent only to reasoning
+models. Unset, and unless a scenario sets its own, Modelpin sends nothing, with one exception:
+`gpt-5.6` (Sol, Terra, Luna), `gpt-6-sol` and `gpt-6-luna` accept tools on Chat Completions only
+with reasoning off, so a scenario that sends tools to them runs with `none`. `last-report.md`
+(the pull-request comment) and `migration-report.md` state the effort each model was sent.
+Testing tool use on those models at another effort needs OpenAI's Responses API, which
+Modelpin does not support yet.
+
 **Cross-vendor:** the baseline and the candidate can be on different providers — record the
 baseline with one `--provider`, check with another (`modelpin check --provider google --to
 gemini-3.1-flash-lite`). `check` reads the baseline from disk, so only the candidate's key is needed
@@ -380,6 +403,7 @@ current model, once.
 |---|---|
 | `modelpin init [dir]` | Write `modelpin.yaml` and a starter scenario, configured from the models your code calls. `--demo` writes the offline sandbox; `--agent-example` adds a tool-calling agent scenario. Never overwrites. |
 | `modelpin scan [path]` | List the model ids a repository (or a single file) uses, and where. |
+| `modelpin watch` | Which of the models in `modelpin.yaml` (its `models:` and `judge_model`; with `--scan`, the ids in your source; your recorded baselines are listed but never alarmed on) are retiring, when, the successor the vendor names, and the exact `check` command; every date carries its source page and fetch date. Exit `0` all clear, `1` a model is inside its notice window (a vendor deprecation, or a shutdown date within 90 days) or already retired, `3` a model is unknown to the registry (not a clearance; ids served by OpenAI-compatible hosts are unknown by design, since the registry reads the three labs' own pages), `4` nothing declared, or the config or registry could not be read. `--json` for scripts; `--registry <path>` to use a different `data/models.json` in place of the shipped one. Never touches the network. |
 | `modelpin draft <file>` | Draft scenarios from one file of your app into `scenarios/.drafts/` for review (one model call). `--count`, `--model`, `--provider`. |
 | `modelpin baseline` | Run every scenario N times on your current model and save the results. |
 | `modelpin check --to <model>` | Replay on a candidate, compare with the baseline, print verdicts, write the report, exit `0`/`1`/`3`/`4`. |
@@ -455,9 +479,15 @@ borderline change rather than raise a false alarm. Know the trade-offs:
   unmeasured.
 - **It measures change, not quality.** It never says one model is better.
 
-**What happened when it was run at scale, live:** 972 same-model checks across six Gemini
-models raised **0** false alarms, and across six real Gemini upgrades it flagged 28 regressions,
-every one of which an independent model rated a material behavior change —
+**What happened when it was run at scale, live:** 972 same-model scenario-checks across six
+Gemini models raised **0** false alarms. They are 82 scenarios re-used across models and
+repeats, not 972 independent trials, and many of those scenarios answered so consistently at
+these settings that no signal could have fired (a harness run on two of the same models put
+that at 230 of 240 trials). So this says the tool did not cry wolf in ordinary use on this
+suite; it is not a false-positive rate, and the bounds are in `docs/fp-measurement.md`.
+Across six real Gemini upgrades it flagged 28
+regressions, and a second language model asked to judge materiality rated every one a
+material behavior change, which is an opinion, not ground truth —
 [docs/live-validation.md](https://github.com/samarthputhraya/modelpin/blob/main/docs/live-validation.md),
 with the bugs that campaign found in Modelpin itself.
 

{modelpin-0.4.2 → modelpin-0.5.2}/README.md

@@ -42,8 +42,17 @@ OK 6 scenario(s) unchanged
   Modelpin or any third party other than the model providers you configure (and, in CI, your own
   pull request).
 
-Modelpin does not watch provider release or retirement feeds and does not open pull requests on
-its own: you choose the candidate model, and a scheduled workflow can re-check it on your clock.
+`modelpin watch` crosses the models your config names (and, with `--scan`, the ones in your
+source) with a registry shipped inside each release, and tells you which of the ones **it knows**
+are retiring, when, the successor the vendor names, and the exact check to run next. Every entry
+in that registry carries the vendor page it was read from and the day it was read; a model the
+registry has never heard of is reported as unknown, never as clear. The registry is transcribed
+from OpenAI, Anthropic and Google's own deprecation and model pages, not from the catalogues
+of OpenAI-compatible
+hosts. **Nothing in Modelpin polls provider feeds**: the registry is refreshed when a release
+ships, so it is only as current as the release you installed (`--registry <path>` points at a
+newer file). Modelpin does not open pull requests on its own: you choose the candidate model,
+and a scheduled workflow can re-check it on your clock.
 
 CLI: `modelpin` (alias `mp`). License: Apache-2.0.
 
@@ -71,7 +80,7 @@ Python 3.11 or newer.
 
 ```bash
 pip install "modelpin[providers]"      # or: pipx install "modelpin[providers]"
-modelpin version                        # -> modelpin 0.4.2
+modelpin version                        # -> modelpin 0.5.2
 ```
 
 The `providers` extra installs the OpenAI, Anthropic and Google SDKs. Plain `pip install modelpin`
@@ -110,7 +119,10 @@ You will see one scenario per verdict:
 
 The command exits **1** because regressions were found — that is what fails a CI build. The full
 report is written to `.modelpin/last-report.md`, the same file the GitHub Action posts on a pull
-request. Edit `traces.json` and re-run to watch the verdicts move.
+request. Beside it, `.modelpin/migration-report.md` tells the same result in plain English for
+whoever decides on the switch: a one-paragraph verdict, each changed scenario with a before and
+after example, what the check did not cover, and the run's date, models, runs and calls. Edit
+`traces.json` and re-run to watch the verdicts move.
 
 ## Use it on your app
 
@@ -290,7 +302,7 @@ key, is not recognised.
 
 | Provider (`providers:` / `--provider`) | Credentials | Notes |
 |---|---|---|
-| `openai` | `OPENAI_API_KEY` | Chat Completions, multi-turn tool calls. Reasoning models (`o1`/`o3`/`o4`/`gpt-5*`) get `max_completion_tokens` and no temperature |
+| `openai` | `OPENAI_API_KEY` | Chat Completions, multi-turn tool calls. Reasoning models (`o1`/`o3`/`o4`/`gpt-5*`/`gpt-6*`) get `max_completion_tokens` and no temperature |
 | `anthropic` | `ANTHROPIC_API_KEY`, **or** Claude on Vertex AI: `ANTHROPIC_VERTEX_PROJECT_ID` + `gcloud auth application-default login` (`CLOUD_ML_REGION` defaults to `global`) | Messages API, multi-turn tool calls. Newer Claude models accept only default sampling, so a scenario's `temperature` is not sent to them. Verified offline against the SDK's request shapes; not yet validated by a live run |
 | `google` | `GEMINI_API_KEY` (AI Studio), **or** Vertex AI: `GOOGLE_GENAI_USE_VERTEXAI=true` + `GOOGLE_CLOUD_PROJECT` + application-default login | Multi-turn tool calls. On Vertex, `GOOGLE_CLOUD_LOCATION` defaults to `global`; in the project's testing `gemini-3.x` ids were not served on `us-central1`, so set it only if you need a specific region |
 | `groq`, `openrouter`, `together`, `cerebras` | `GROQ_API_KEY`, `OPENROUTER_API_KEY`, `TOGETHER_API_KEY`, `CEREBRAS_API_KEY` | OpenAI-compatible endpoints. Groq has a free tier |
@@ -301,6 +313,17 @@ vendor (`gpt-*`, `claude-*` and `gemini-*` do; `openai/gpt-oss-120b` on Groq doe
 model that is neither the one you run today nor the candidate. Remove `judge_model:` to compare
 only tool calls, refusals and text checks, with no extra calls.
 
+**Reasoning effort** (OpenAI reasoning models): set `reasoning_effort:` in `modelpin.yaml`, or
+pass `--reasoning-effort` to `baseline` and `check` (both read it), to measure the effort you
+will ship (`none`, `minimal`, `low`, `medium`, `high` or `xhigh`; which values a model accepts
+is OpenAI's call). A scenario can set its own `reasoning_effort:`. It is sent only to reasoning
+models. Unset, and unless a scenario sets its own, Modelpin sends nothing, with one exception:
+`gpt-5.6` (Sol, Terra, Luna), `gpt-6-sol` and `gpt-6-luna` accept tools on Chat Completions only
+with reasoning off, so a scenario that sends tools to them runs with `none`. `last-report.md`
+(the pull-request comment) and `migration-report.md` state the effort each model was sent.
+Testing tool use on those models at another effort needs OpenAI's Responses API, which
+Modelpin does not support yet.
+
 **Cross-vendor:** the baseline and the candidate can be on different providers — record the
 baseline with one `--provider`, check with another (`modelpin check --provider google --to
 gemini-3.1-flash-lite`). `check` reads the baseline from disk, so only the candidate's key is needed
@@ -337,6 +360,7 @@ current model, once.
 |---|---|
 | `modelpin init [dir]` | Write `modelpin.yaml` and a starter scenario, configured from the models your code calls. `--demo` writes the offline sandbox; `--agent-example` adds a tool-calling agent scenario. Never overwrites. |
 | `modelpin scan [path]` | List the model ids a repository (or a single file) uses, and where. |
+| `modelpin watch` | Which of the models in `modelpin.yaml` (its `models:` and `judge_model`; with `--scan`, the ids in your source; your recorded baselines are listed but never alarmed on) are retiring, when, the successor the vendor names, and the exact `check` command; every date carries its source page and fetch date. Exit `0` all clear, `1` a model is inside its notice window (a vendor deprecation, or a shutdown date within 90 days) or already retired, `3` a model is unknown to the registry (not a clearance; ids served by OpenAI-compatible hosts are unknown by design, since the registry reads the three labs' own pages), `4` nothing declared, or the config or registry could not be read. `--json` for scripts; `--registry <path>` to use a different `data/models.json` in place of the shipped one. Never touches the network. |
 | `modelpin draft <file>` | Draft scenarios from one file of your app into `scenarios/.drafts/` for review (one model call). `--count`, `--model`, `--provider`. |
 | `modelpin baseline` | Run every scenario N times on your current model and save the results. |
 | `modelpin check --to <model>` | Replay on a candidate, compare with the baseline, print verdicts, write the report, exit `0`/`1`/`3`/`4`. |
@@ -412,9 +436,15 @@ borderline change rather than raise a false alarm. Know the trade-offs:
   unmeasured.
 - **It measures change, not quality.** It never says one model is better.
 
-**What happened when it was run at scale, live:** 972 same-model checks across six Gemini
-models raised **0** false alarms, and across six real Gemini upgrades it flagged 28 regressions,
-every one of which an independent model rated a material behavior change —
+**What happened when it was run at scale, live:** 972 same-model scenario-checks across six
+Gemini models raised **0** false alarms. They are 82 scenarios re-used across models and
+repeats, not 972 independent trials, and many of those scenarios answered so consistently at
+these settings that no signal could have fired (a harness run on two of the same models put
+that at 230 of 240 trials). So this says the tool did not cry wolf in ordinary use on this
+suite; it is not a false-positive rate, and the bounds are in `docs/fp-measurement.md`.
+Across six real Gemini upgrades it flagged 28
+regressions, and a second language model asked to judge materiality rated every one a
+material behavior change, which is an opinion, not ground truth —
 [docs/live-validation.md](https://github.com/samarthputhraya/modelpin/blob/main/docs/live-validation.md),
 with the bugs that campaign found in Modelpin itself.