model-eol 0.5.2 → 0.6.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +344 -230
- package/SPEC.md +12 -1
- package/bot/bot.mjs +19 -6
- package/bot/lib/common.mjs +6 -0
- package/feeds/anthropic.json +47 -11
- package/feeds/google.json +26 -3
- package/feeds/openai.json +4 -4
- package/lib/config.mjs +61 -3
- package/lib/feeds.mjs +36 -6
- package/lib/plan.mjs +7 -0
- package/lib/reports.mjs +80 -16
- package/lib/scanner.mjs +2 -0
- package/lib/validate-document.mjs +1 -1
- package/lib/validate-feed.mjs +22 -3
- package/package.json +1 -1
- package/schema/model-eol.bot-config.schema.json +28 -1
- package/schema/model-eol.check.schema.json +2 -1
- package/schema/model-eol.inventory.schema.json +13 -1
- package/schema/model-eol.plan.schema.json +13 -0
- package/schema/model-eol.schema.json +2 -2
package/README.md
CHANGED
|
@@ -17,37 +17,41 @@ tooling that turns it into a Dependabot for models.**
|
|
|
17
17
|
## The problem
|
|
18
18
|
|
|
19
19
|
On July 23, 2026, OpenAI shut down a scheduled wave of model snapshots. One of my
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
my toolchain could not
|
|
23
|
-
at install, Dependabot opens the PR, and an advisory can fail CI. A model's
|
|
24
|
-
retirement date usually lives in an HTML docs page and an email.
|
|
20
|
+
research workflows failed that morning with a model-not-found error. The
|
|
21
|
+
retirement had been documented for months. I had not seen it, because a model ID
|
|
22
|
+
was the one dependency my toolchain could not track.
|
|
25
23
|
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
24
|
+
Other dependencies do not work this way. When an npm package is deprecated, the
|
|
25
|
+
warning prints at install, Dependabot opens a PR, and an advisory can fail CI. A
|
|
26
|
+
model's retirement date usually lives in an HTML docs page and an email.
|
|
27
|
+
|
|
28
|
+
Ordinary software already solved this layer: endoflife.date for end-of-life
|
|
29
|
+
dates, OSV for vulnerability feeds. This project is the model version. The
|
|
30
|
+
schema comes first and the tooling second, so existing trackers can share one
|
|
31
|
+
format instead of each scraping the same pages.
|
|
29
32
|
|
|
30
33
|
## What's here
|
|
31
34
|
|
|
32
35
|
- **`SPEC.md`** - the feed format (model-eol/0.1): id, aliases, announced, shutdown,
|
|
33
36
|
replacement, `distributions` for per-channel lifecycles, and publisher `policy`
|
|
34
|
-
floors.
|
|
37
|
+
floors. It is small enough that a provider could serve it at
|
|
35
38
|
`/.well-known/model-eol.json` in an afternoon.
|
|
36
|
-
- **`feeds/`** - <!-- feeds-status -->Amazon (4 entries), Anthropic (
|
|
37
|
-
dated entry carries a source URL.
|
|
39
|
+
- **`feeds/`** - <!-- feeds-status -->Amazon (4 entries), Anthropic (30 entries), Google (93 entries) and OpenAI (194 entries), generated from the providers' live deprecation pages plus the AWS Bedrock and Google Vertex AI lifecycle pages, feed data generated 2026-09-03<!-- /feeds-status -->. Every
|
|
40
|
+
dated entry carries a source URL. Anthropic's "not sooner than" dates for
|
|
41
|
+
active models are included as `tentative` planning floors.
|
|
38
42
|
- **`check.mjs`** - zero-dependency CLI: CI gate, PR diff gate, inventory, CycloneDX
|
|
39
43
|
ML-BOM export, retirement schedule, alerts and badges, migration plan/apply,
|
|
40
44
|
and public document validation.
|
|
41
45
|
- **`refresh/`** - regenerates the feeds from provider pages, models endpoints, and
|
|
42
46
|
distributor lifecycle pages, with a semantic diff for human review.
|
|
43
|
-
- **`bot/`** - the Dependabot part: a
|
|
44
|
-
migration PR or issue per retiring model
|
|
47
|
+
- **`bot/`** - the Dependabot part: a scheduled GitHub workflow that maintains one
|
|
48
|
+
migration PR or issue per retiring model. It ships as the `model-eol-bot`
|
|
45
49
|
binary in the same npm package.
|
|
46
50
|
- **Public contract** - versioned schemas, hosted feeds, refresh health, a
|
|
47
|
-
publication manifest, and an Atom changelog
|
|
51
|
+
publication manifest, and an Atom changelog at
|
|
48
52
|
[`thossullivan.github.io/model-eol`](https://thossullivan.github.io/model-eol/).
|
|
49
53
|
Publication requires a refresh receipt for the exact feeds on `main`, followed
|
|
50
|
-
by an exact-byte check
|
|
54
|
+
by an exact-byte check of every deployed asset.
|
|
51
55
|
|
|
52
56
|
## Try it
|
|
53
57
|
|
|
@@ -57,8 +61,8 @@ One command, no install, no API keys:
|
|
|
57
61
|
npx model-eol path/to/your/repo --days 90
|
|
58
62
|
```
|
|
59
63
|
|
|
60
|
-
That is the whole CI gate.
|
|
61
|
-
identical, because there are no dependencies to install either way:
|
|
64
|
+
That is the whole CI gate. If you prefer not to use npm, clone and run. The
|
|
65
|
+
result is identical, because there are no dependencies to install either way:
|
|
62
66
|
|
|
63
67
|
```sh
|
|
64
68
|
git clone https://github.com/thossullivan/model-eol && cd model-eol
|
|
@@ -102,25 +106,25 @@ Sample output against the test fixture (2026-08-01):
|
|
|
102
106
|
earliest risk: 2026-07-23 (o3-deep-research-2025-06-26)
|
|
103
107
|
```
|
|
104
108
|
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
CI
|
|
109
|
+
The command exits 1 when any finding is at or past the threshold, so it works
|
|
110
|
+
in CI as-is. The badge JSON works with the
|
|
111
|
+
[shields endpoint](https://shields.io/endpoint) from any URL your CI can reach.
|
|
108
112
|
|
|
109
113
|
**No API keys, no accounts.** Scanning is static analysis and the feeds are data
|
|
110
|
-
files
|
|
111
|
-
credentials. The bot needs
|
|
112
|
-
PRs to trigger checks
|
|
113
|
-
|
|
114
|
-
|
|
114
|
+
files. The checker, PR gate, schedule, SBOM export, and badge run with no
|
|
115
|
+
credentials at all. The bot needs a GitHub token, and a fine-grained PAT if you
|
|
116
|
+
want its PRs to trigger checks. Provider API keys are optional in exactly two
|
|
117
|
+
places: your own eval command, and the models-endpoint coverage in the feed
|
|
118
|
+
refresh.
|
|
115
119
|
|
|
116
|
-
Both command-line binaries ship in one zero-dependency npm
|
|
117
|
-
suite packs that
|
|
118
|
-
|
|
119
|
-
peer, or optional npm
|
|
120
|
+
Both command-line binaries ship in one zero-dependency npm package. The test
|
|
121
|
+
suite packs that package, installs it into an empty directory with the network
|
|
122
|
+
disabled, and runs both binaries. The suite also fails if any runtime,
|
|
123
|
+
development, peer, or optional npm dependency is added.
|
|
120
124
|
|
|
121
125
|
## Use it as a GitHub Action
|
|
122
126
|
|
|
123
|
-
The same gate as a composite action
|
|
127
|
+
The same gate as a composite action. No npm install, feeds bundled, pinned to
|
|
124
128
|
the `v0` line:
|
|
125
129
|
|
|
126
130
|
```yaml
|
|
@@ -139,26 +143,30 @@ the `v0` line:
|
|
|
139
143
|
scope: direct
|
|
140
144
|
```
|
|
141
145
|
|
|
142
|
-
|
|
143
|
-
or retiring within the threshold. Add `via: aws-bedrock` (or
|
|
144
|
-
to judge by a distributor's clock.
|
|
145
|
-
|
|
146
|
+
This is the PR gate. It fails only when *this* change adds a model that is
|
|
147
|
+
retired or retiring within the threshold. Add `via: aws-bedrock` (or
|
|
148
|
+
`azure-ai-foundry`) to judge by a distributor's clock.
|
|
149
|
+
|
|
150
|
+
A copy-paste workflow with both gates, the PR diff and a weekly full-repository
|
|
151
|
+
check, is in
|
|
146
152
|
[`examples/workflows/model-eol.yml`](examples/workflows/model-eol.yml). The
|
|
147
|
-
Action also exposes `inventory`, `schedule`, `alert`, `plan`, and `validate`.
|
|
148
|
-
`format: cyclonedx` for an ML-BOM
|
|
149
|
-
workflow artifact while
|
|
153
|
+
Action also exposes `inventory`, `schedule`, `alert`, `plan`, and `validate`.
|
|
154
|
+
Use `format: cyclonedx` for an ML-BOM. Use `output-file` to keep any report as
|
|
155
|
+
a workflow artifact while the same output still prints in the job log.
|
|
150
156
|
|
|
151
|
-
CycloneDX exports
|
|
152
|
-
lifecycle channel.
|
|
153
|
-
|
|
154
|
-
under deterministic
|
|
155
|
-
and
|
|
157
|
+
CycloneDX exports contain one machine-learning component per canonical model
|
|
158
|
+
and lifecycle channel. A direct publisher reference (`publisher-direct`), an
|
|
159
|
+
Azure reference, and a Bedrock reference to the same model therefore keep their
|
|
160
|
+
own status and shutdown date, under deterministic channel-qualified `bom-ref`
|
|
161
|
+
values. Occurrences are sorted and attached to the component whose clock judged
|
|
162
|
+
them.
|
|
156
163
|
|
|
157
164
|
## One repository policy
|
|
158
165
|
|
|
159
|
-
The CLI, Action, and bot all
|
|
160
|
-
Git repository
|
|
161
|
-
`--config FILE` selects an explicit policy when
|
|
166
|
+
The CLI, the Action, and the bot all read a strict `.model-eol.json` at the
|
|
167
|
+
root of the target Git repository. CLI flags and Action inputs override
|
|
168
|
+
configured values. `--config FILE` selects an explicit policy when discovery
|
|
169
|
+
is ambiguous:
|
|
162
170
|
|
|
163
171
|
```json
|
|
164
172
|
{
|
|
@@ -169,41 +177,93 @@ Git repository root. CLI flags or Action inputs override configured values, and
|
|
|
169
177
|
"models": ["o3-deep-research"],
|
|
170
178
|
"paths": ["test/fixtures/**", "vendor"]
|
|
171
179
|
},
|
|
180
|
+
"waivers": [
|
|
181
|
+
{
|
|
182
|
+
"model": "gpt-4",
|
|
183
|
+
"paths": ["services/legacy/**"],
|
|
184
|
+
"via": "aws-bedrock",
|
|
185
|
+
"reason": "Vendor contract pins this model until the Q1 migration.",
|
|
186
|
+
"owner": "@platform-team",
|
|
187
|
+
"expires": "2026-12-31"
|
|
188
|
+
}
|
|
189
|
+
],
|
|
172
190
|
"issues": { "enabled": true },
|
|
173
191
|
"eval": { "command": "node scripts/model-eol-eval.mjs" }
|
|
174
192
|
}
|
|
175
193
|
```
|
|
176
194
|
|
|
177
|
-
Unknown keys and invalid values
|
|
178
|
-
in `ignore.models` suppresses the whole alias family
|
|
195
|
+
Unknown keys and invalid values are errors. A canonical model ID or any alias
|
|
196
|
+
in `ignore.models` suppresses the whole alias family. `ignore.paths` takes
|
|
179
197
|
repo-relative `*`, `**`, and `?` globs and excludes matching files before scan
|
|
180
|
-
coverage
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
|
|
185
|
-
|
|
186
|
-
Draft-07 cannot
|
|
187
|
-
|
|
188
|
-
|
|
189
|
-
|
|
190
|
-
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
|
|
194
|
-
|
|
195
|
-
|
|
196
|
-
|
|
197
|
-
|
|
198
|
-
|
|
199
|
-
|
|
200
|
-
|
|
198
|
+
coverage is counted.
|
|
199
|
+
|
|
200
|
+
With no config, the CLI keeps its historical `scope: all`. Once a config exists,
|
|
201
|
+
omitted values use the bot defaults: `days: 90`, `scope: direct`, and the
|
|
202
|
+
publisher clock. The portable structure is defined by
|
|
203
|
+
[`schema/model-eol.bot-config.schema.json`](schema/model-eol.bot-config.schema.json).
|
|
204
|
+
`model-eol validate` also enforces the runtime rules that Draft-07 cannot
|
|
205
|
+
express.
|
|
206
|
+
|
|
207
|
+
### Ignores and waivers
|
|
208
|
+
|
|
209
|
+
Ignores and waivers answer different questions.
|
|
210
|
+
|
|
211
|
+
`ignore` hides inventory noise: fixtures, caches, and vendored catalogs that are
|
|
212
|
+
not live dependencies. Ignored references do not appear in reports.
|
|
213
|
+
|
|
214
|
+
A **waiver** records a deliberate exception on a live dependency: which model,
|
|
215
|
+
why, who owns it, and the date it expires. A waiver never hides its finding.
|
|
216
|
+
The reference stays in `check`, `inventory`, `schedule`, `alert`, `plan`, and
|
|
217
|
+
the CycloneDX export, with its reason, owner, expiry, and active state attached.
|
|
218
|
+
The `waiver` field appears on a finding only when a waiver matched it:
|
|
219
|
+
|
|
220
|
+
```
|
|
221
|
+
✗ app.py:2 o3-deep-research RETIRED 2026-07-23 (42 days ago) -> gpt-5.6-sol [waived until 2026-12-31 by @research-platform: Research pipeline pinned until the Q4 rerun.]
|
|
222
|
+
```
|
|
223
|
+
|
|
224
|
+
While a waiver is active, that finding is left out of the exit-1 total, the
|
|
225
|
+
badge counts, and the alert errors, and the bot opens no migration work for it.
|
|
226
|
+
On the expiry date the finding is actionable again, and the report names the
|
|
227
|
+
waiver that lapsed and who owned it. Expiry uses UTC dates, and the expiry date
|
|
228
|
+
itself counts as expired.
|
|
229
|
+
|
|
230
|
+
The waiver fields work like this:
|
|
231
|
+
|
|
232
|
+
- `model` matches the whole alias family.
|
|
233
|
+
- `paths` takes the same repo-relative globs as `ignore.paths`.
|
|
234
|
+
- `via` narrows the waiver to one clock: a distributor id, or `publisher` for
|
|
235
|
+
the direct publisher clock. Omit it to waive every clock.
|
|
236
|
+
- The first active matching waiver wins.
|
|
237
|
+
- A policy holds at most 500 waivers across the root and its overrides.
|
|
238
|
+
|
|
239
|
+
A malformed waiver, such as a missing owner, a relative duration, or an unknown
|
|
240
|
+
key, is a config error like any other. An expired waiver is not an error. It is
|
|
241
|
+
simply no longer active. The CycloneDX export records each component's waiver
|
|
242
|
+
state as active, partial, or expired.
|
|
243
|
+
|
|
244
|
+
### Coverage limits
|
|
245
|
+
|
|
246
|
+
For content-heavy repositories, start with `inventory`. Then exclude caches,
|
|
247
|
+
generated catalogs, archived fixtures, and documentation that are not live
|
|
248
|
+
model dependencies. The published-repository tests verify both outcomes: an
|
|
249
|
+
unconfigured scan reports the evidence, and an explicit path policy produces a
|
|
250
|
+
clean check and an empty bot decision table without changing repository
|
|
251
|
+
content.
|
|
252
|
+
|
|
253
|
+
Scans do not recurse into tracked Git submodules. A detected gitlink is
|
|
254
|
+
reported as incomplete coverage, so `check` and `plan` fail unless the path is
|
|
255
|
+
ignored or `--allow-incomplete` is set. To inspect a submodule, run model-eol
|
|
256
|
+
against its checkout as a separate target with its own policy.
|
|
257
|
+
|
|
258
|
+
Untracked nested Git repositories work the same way. The parent reports
|
|
201
259
|
`nested-repository-skipped` and never recurses into the nested checkout. Ignore
|
|
202
|
-
an intentional nested repository
|
|
203
|
-
|
|
260
|
+
an intentional nested repository, allow incomplete coverage, or scan it
|
|
261
|
+
separately.
|
|
204
262
|
|
|
205
|
-
|
|
206
|
-
|
|
263
|
+
### Monorepos
|
|
264
|
+
|
|
265
|
+
Mixed-provider monorepos keep the top-level values as defaults, then apply path
|
|
266
|
+
policies and a lifecycle channel per reference:
|
|
207
267
|
|
|
208
268
|
```json
|
|
209
269
|
{
|
|
@@ -225,126 +285,173 @@ then apply path policies and a lifecycle channel per reference:
|
|
|
225
285
|
}
|
|
226
286
|
```
|
|
227
287
|
|
|
228
|
-
Matching overrides apply in order
|
|
229
|
-
|
|
230
|
-
|
|
231
|
-
|
|
232
|
-
|
|
233
|
-
|
|
234
|
-
|
|
235
|
-
|
|
236
|
-
|
|
237
|
-
|
|
238
|
-
|
|
239
|
-
|
|
240
|
-
|
|
241
|
-
|
|
242
|
-
|
|
243
|
-
|
|
244
|
-
|
|
245
|
-
|
|
246
|
-
|
|
247
|
-
|
|
248
|
-
|
|
249
|
-
|
|
250
|
-
|
|
251
|
-
|
|
252
|
-
|
|
253
|
-
|
|
254
|
-
|
|
288
|
+
Matching overrides apply in order. Later scalar values win. Ignore and waiver
|
|
289
|
+
lists add up, so an override can carry path-scoped `waivers` with the same shape
|
|
290
|
+
as the root. The last matching route wins. An exact `match`/`model` pair
|
|
291
|
+
resolves a deployment alias without making cloud or gateway references
|
|
292
|
+
patchable. Explicit CLI flags and Action inputs still take precedence.
|
|
293
|
+
|
|
294
|
+
JSON inventory and plan output record each reference's effective threshold,
|
|
295
|
+
scope, requested channel, and matching rule indexes. A Bedrock service and a
|
|
296
|
+
direct publisher call in the same repository therefore never share one global
|
|
297
|
+
clock by accident.
|
|
298
|
+
|
|
299
|
+
## One model, several retirement dates
|
|
300
|
+
|
|
301
|
+
The same model retires on different dates on different channels. A checker that
|
|
302
|
+
ignores this is wrong in both directions. o3-deep-research retired at OpenAI on
|
|
303
|
+
July 23 but stays available on Azure AI Foundry until December 26.
|
|
304
|
+
claude-opus-4-1 retires at Anthropic on August 5 but stays on Amazon Bedrock
|
|
305
|
+
until January 8, 2027.
|
|
306
|
+
|
|
307
|
+
`distributions` in the spec carries these per-channel dates. `--via <distributor>`
|
|
308
|
+
judges your repo by the channel you call. The distributor refresh keeps the
|
|
309
|
+
Bedrock and Vertex dates current from their lifecycle pages. A distributor's
|
|
310
|
+
later date gives you more time for the same migration. It is not a reason to
|
|
311
|
+
skip it.
|
|
312
|
+
|
|
313
|
+
## Policy floors
|
|
314
|
+
|
|
315
|
+
A feed entry with no `announced` and no `shutdown` means "nothing scheduled as
|
|
316
|
+
of `generated`". That is a positive statement, and it is one that scraping HTML
|
|
317
|
+
cannot make on its own.
|
|
318
|
+
|
|
319
|
+
Publishers with a stated minimum notice period carry a `policy` floor. Anthropic
|
|
320
|
+
states at least 60 days. The schedule uses that floor to compute a
|
|
321
|
+
no-earlier-than planning date from the feed refresh. This is stated policy, not
|
|
322
|
+
a contract. OpenAI publishes no formal notice period, so its feed makes no
|
|
323
|
+
forward claim.
|
|
324
|
+
|
|
325
|
+
Anthropic also lists a tentative retirement date, "not sooner than", for every
|
|
326
|
+
active model. The feed carries those as `shutdown` with
|
|
327
|
+
`date_precision: "tentative"` and no `announced` date, because nothing has been
|
|
328
|
+
announced. The checker treats a tentative date as a floor, not a schedule. The
|
|
329
|
+
finding is `scheduled`, never `retiring` or `retired`. `safe_until` is the later
|
|
330
|
+
of the floor and the policy date. A tentative floor cannot fail CI, and the bot
|
|
331
|
+
opens no work for it. It does appear in every schedule, so planning can start
|
|
332
|
+
before the announcement:
|
|
333
|
+
|
|
334
|
+
```
|
|
335
|
+
· app.py:1 claude-sonnet-4-5-20250929 scheduled - tentative, not announced: not sooner than 2026-09-29, guaranteed until 2026-10-17 by anthropic policy
|
|
336
|
+
```
|
|
337
|
+
|
|
338
|
+
When Anthropic announces the retirement, the announcement wins. The entry
|
|
339
|
+
becomes an ordinary exact date with an `announced` date.
|
|
340
|
+
|
|
341
|
+
`earliest` is different. Google's "earliest possible" shutdown dates are
|
|
342
|
+
announced deprecations, and the checker treats them as the scheduled date.
|
|
343
|
+
|
|
344
|
+
A repository that uses many active Claude models gets one floor line per
|
|
345
|
+
reference. To read only announced retirements, filter on `date_precision`:
|
|
346
|
+
|
|
347
|
+
```sh
|
|
348
|
+
node check.mjs schedule path/to/your/repo --json \
|
|
349
|
+
| jq '.items |= map(select(.date_precision != "tentative"))'
|
|
350
|
+
```
|
|
255
351
|
|
|
256
352
|
## Direct-first inventory
|
|
257
353
|
|
|
258
|
-
Direct API usage
|
|
259
|
-
scanned and checked immediately. Cloud providers and
|
|
260
|
-
Bedrock, Vertex, OpenRouter, LiteLLM,
|
|
261
|
-
|
|
262
|
-
|
|
354
|
+
Direct API usage, meaning OpenAI, Anthropic, Gemini, or similar SDK calls with
|
|
355
|
+
real model IDs, is scanned and checked immediately. Cloud providers and
|
|
356
|
+
gateways such as Azure Foundry, Bedrock, Vertex, OpenRouter, LiteLLM, and
|
|
357
|
+
Portkey often hide the model behind a deployment name or a routing alias. Those
|
|
358
|
+
references stay in the inventory as resolver targets, because static analysis
|
|
359
|
+
cannot know which model is deployed behind them:
|
|
263
360
|
|
|
264
361
|
```
|
|
265
362
|
? src/ai.ts:12 azure-ai-foundry Azure Foundry/OpenAI deployment reference
|
|
266
363
|
? src/ai.ts:30 openrouter OpenRouter gateway reference
|
|
267
364
|
```
|
|
268
365
|
|
|
269
|
-
Use `--scope direct` to fail CI only on what static analysis can prove.
|
|
270
|
-
strings absent from the feeds
|
|
271
|
-
stay visible without
|
|
272
|
-
|
|
273
|
-
The scanner covers code and configuration files
|
|
274
|
-
|
|
275
|
-
Ruby, Go, Java, C#, Rust, C/C++,
|
|
276
|
-
Objective-C, including `.baml`
|
|
277
|
-
|
|
278
|
-
|
|
279
|
-
|
|
280
|
-
they carry explicit model-eol generator
|
|
281
|
-
`.tfvars`) are scanned
|
|
366
|
+
Use `--scope direct` to fail CI only on what static analysis can prove.
|
|
367
|
+
Model-like strings that are absent from the feeds are reported as non-failing
|
|
368
|
+
**candidates**, so feed gaps stay visible without false alarms.
|
|
369
|
+
|
|
370
|
+
The scanner covers code and configuration files. The list in
|
|
371
|
+
[`CODE_EXT`](lib/scanner.mjs) is the source of truth: Python,
|
|
372
|
+
TypeScript/JavaScript, JSON/YAML/TOML, shell, Ruby, Go, Java, C#, Rust, C/C++,
|
|
373
|
+
PHP, Swift, Kotlin, Terraform, SQL, Scala, and Objective-C, including `.baml`
|
|
374
|
+
sources. Publisher-compatible BAML provider/model pairs can be plan targets.
|
|
375
|
+
Generated BAML clients and sources marked `DO NOT EDIT` or `@generated` are
|
|
376
|
+
skipped. Model-eol's own JSON reports are skipped on reruns. Third-party
|
|
377
|
+
CycloneDX files are scanned unless they carry explicit model-eol generator
|
|
378
|
+
provenance. Terraform files (`.tf`, `.tfvars`) are scanned, and `.terraform`
|
|
379
|
+
directories are excluded.
|
|
282
380
|
|
|
283
381
|
## The bot
|
|
284
382
|
|
|
285
|
-
Copy `bot.yml.example` to `.github/workflows/model-eol-bot.yml` for a weekly
|
|
286
|
-
|
|
287
|
-
context in the body
|
|
288
|
-
|
|
289
|
-
|
|
290
|
-
|
|
291
|
-
|
|
292
|
-
|
|
293
|
-
|
|
294
|
-
|
|
383
|
+
Copy `bot.yml.example` to `.github/workflows/model-eol-bot.yml` for a weekly
|
|
384
|
+
run. It maintains exactly one labelled migration PR per retiring model, with
|
|
385
|
+
the full feed context in the body and never auto-merged, and one issue per
|
|
386
|
+
finding that needs a human.
|
|
387
|
+
|
|
388
|
+
The workflow resolves `model-eol@0` from npm once and uses that exact version
|
|
389
|
+
across its plan, read-only evaluation, and write-token publication jobs.
|
|
390
|
+
Consumer repositories do not copy `check.mjs` or the `bot/` source directory.
|
|
391
|
+
|
|
392
|
+
Dismissals are respected. A reintroduced retired model creates fresh work.
|
|
393
|
+
Bot-owned PRs and issues close when their finding disappears or is waived. When
|
|
394
|
+
a waiver expires, the bot treats the finding as new rather than inheriting the
|
|
395
|
+
earlier dismissal.
|
|
396
|
+
|
|
397
|
+
In a repository without a configured eval, preview everything with no GitHub
|
|
398
|
+
calls and no project dependency:
|
|
295
399
|
|
|
296
400
|
```sh
|
|
297
401
|
npx --yes --package=model-eol@0 model-eol-bot --dry-run --target-dir . --repo OWNER/REPO
|
|
298
402
|
```
|
|
299
403
|
|
|
300
|
-
The workflow honors configured `days`, `scope`, and `via` values
|
|
301
|
-
it uses the bot defaults of 90 days and direct references. `eval.command`
|
|
302
|
-
comes from the config unless the `MODEL_EOL_EVAL_COMMAND` repository
|
|
303
|
-
set as an explicit
|
|
304
|
-
|
|
305
|
-
|
|
306
|
-
|
|
307
|
-
|
|
404
|
+
The workflow honors configured `days`, `scope`, and `via` values. Without a
|
|
405
|
+
config it uses the bot defaults of 90 days and direct references. `eval.command`
|
|
406
|
+
also comes from the config, unless the `MODEL_EOL_EVAL_COMMAND` repository
|
|
407
|
+
variable is set as an explicit override.
|
|
408
|
+
|
|
409
|
+
Each patchable model is applied in an isolated checkout and evaluated on its
|
|
410
|
+
own, with its own old and new IDs, timeout, bounded report, and explicitly
|
|
411
|
+
forwarded environment names. The result manifest is bound to its content and
|
|
412
|
+
pinned to the evaluated Git commit. A passing migration can proceed while a
|
|
413
|
+
failing one stays blocked, and nothing is published against a newer,
|
|
308
414
|
unevaluated default branch.
|
|
309
415
|
|
|
310
416
|
### Prove the swap works in your repository
|
|
311
417
|
|
|
312
|
-
Model behavior is repository
|
|
313
|
-
good enough to authorize a migration. Check a harness into the
|
|
314
|
-
name it
|
|
315
|
-
`MODEL_EOL_EVAL_COMMAND` repository variable overrides that command
|
|
316
|
-
the config is authoritative. With neither configured, the result manifest
|
|
317
|
-
that evaluation was not configured.
|
|
418
|
+
Model behavior is specific to your repository, so model-eol never guesses which
|
|
419
|
+
test is good enough to authorize a migration. Check a harness into the
|
|
420
|
+
repository and name it in `.model-eol.json`. A non-empty
|
|
421
|
+
`MODEL_EOL_EVAL_COMMAND` repository variable overrides that command. Otherwise
|
|
422
|
+
the config is authoritative. With neither configured, the result manifest
|
|
423
|
+
records that evaluation was not configured.
|
|
318
424
|
|
|
319
425
|
Start by copying [`examples/model-eol-eval.mjs`](examples/model-eol-eval.mjs) to
|
|
320
|
-
`scripts/model-eol-eval.mjs
|
|
321
|
-
deterministic test that proves the behavior your application relies on. A
|
|
426
|
+
`scripts/model-eol-eval.mjs`. Replace its `VERIFY` command with the smallest
|
|
427
|
+
deterministic test that proves the behavior your application relies on. A good
|
|
322
428
|
harness checks response or tool-call shape, structured-output parsing, required
|
|
323
|
-
capabilities, and a small golden
|
|
324
|
-
evidence, but it does not
|
|
325
|
-
|
|
326
|
-
|
|
327
|
-
|
|
328
|
-
Point `eval:model-eol` at the
|
|
329
|
-
|
|
429
|
+
capabilities, and a small set of golden tasks. A passing build is supporting
|
|
430
|
+
evidence, but it does not prove model behavior by itself.
|
|
431
|
+
|
|
432
|
+
The starter calls a dedicated `npm run eval:model-eol` script, so adopting it
|
|
433
|
+
requires an explicit repository test. That prevents an unrelated unit suite from
|
|
434
|
+
being treated as an LLM-behavior eval by accident. Point `eval:model-eol` at the
|
|
435
|
+
behavior test, not back at the harness.
|
|
330
436
|
|
|
331
|
-
The harness runs after the proposed migration in an isolated checkout and
|
|
437
|
+
The harness runs after the proposed migration, in an isolated checkout, and
|
|
438
|
+
receives:
|
|
332
439
|
|
|
333
440
|
- `MODEL_EOL_OLD_ID` and `MODEL_EOL_NEW_ID` for the current migration.
|
|
334
441
|
- `MODEL_EOL_PLAN`, a one-model plan containing the exact changed references.
|
|
335
442
|
- `MODEL_EOL_REPORT`, the required path for a bounded, publishable Markdown receipt.
|
|
336
443
|
|
|
337
|
-
Exit zero plus a regular report file means pass
|
|
338
|
-
report, tracked workspace
|
|
339
|
-
stdout and stderr are
|
|
340
|
-
in the report
|
|
341
|
-
`eval.pass_env`. That allowlist controls normal subprocess environment
|
|
342
|
-
|
|
343
|
-
|
|
344
|
-
the publication job never
|
|
444
|
+
Exit zero plus a regular report file means pass. Any other exit, a timeout, a
|
|
445
|
+
missing report, a tracked workspace change, or checkout drift fails that
|
|
446
|
+
migration. Test stdout and stderr are not published. Put only non-secret
|
|
447
|
+
evidence in the report. Forward only the credentials the eval code needs,
|
|
448
|
+
through `eval.pass_env`. That allowlist controls normal subprocess environment
|
|
449
|
+
forwarding. It is not an OS sandbox and does not isolate code running as the
|
|
450
|
+
same user. The security boundary is that the evaluation job has no repository
|
|
451
|
+
write token, and the publication job never runs repository-owned eval code.
|
|
345
452
|
|
|
346
453
|
Before enabling the scheduled bot, commit the harness and policy on a clean
|
|
347
|
-
branch and
|
|
454
|
+
branch and run the same isolated evaluator locally:
|
|
348
455
|
|
|
349
456
|
```bash
|
|
350
457
|
set -euo pipefail
|
|
@@ -370,30 +477,29 @@ MODEL_EOL_EVAL_RESULTS_FILE="$tmp/eval.json" \
|
|
|
370
477
|
--dry-run --target-dir "$repo" --repo OWNER/REPO
|
|
371
478
|
```
|
|
372
479
|
|
|
373
|
-
Each planned model is patched and evaluated
|
|
374
|
-
removes the temporary
|
|
375
|
-
after the dry run if you want to
|
|
376
|
-
`bot.yml.example` run the same contract with any
|
|
377
|
-
|
|
378
|
-
|
|
379
|
-
Inline `model-eol-bot --eval` and the
|
|
380
|
-
refused. A configured publication must consume the commit-bound manifest
|
|
381
|
-
`model-eol-bot evaluate
|
|
382
|
-
|
|
383
|
-
On its first actionable run, the bot creates the `model-eol` label. If
|
|
384
|
-
policy prevents label creation or assignment,
|
|
385
|
-
work whose ownership cannot be
|
|
386
|
-
the label, metadata, deterministic branch, repository, base, and Git
|
|
387
|
-
For HTTPS remotes
|
|
388
|
-
temporary Git processes as a host-scoped authorization
|
|
389
|
-
in a remote URL.
|
|
390
|
-
|
|
391
|
-
Two operational notes
|
|
392
|
-
|
|
393
|
-
|
|
394
|
-
|
|
395
|
-
|
|
396
|
-
in the reconciliation job.
|
|
480
|
+
Each planned model is patched and evaluated on its own. The cleanup trap
|
|
481
|
+
removes the temporary files when the shell exits. Copy `eval.json` elsewhere
|
|
482
|
+
after the dry run if you want to keep it as evidence. Then let
|
|
483
|
+
`bot.yml.example` run the same contract, with any provider keys you opt in
|
|
484
|
+
available only to its read-only evaluate job, never to its publish job.
|
|
485
|
+
|
|
486
|
+
Inline `model-eol-bot --eval` and the older unbound report and status artifacts
|
|
487
|
+
are refused. A configured publication must consume the commit-bound manifest
|
|
488
|
+
from `model-eol-bot evaluate`. Missing results fail before any GitHub API call.
|
|
489
|
+
|
|
490
|
+
On its first actionable run, the bot creates the `model-eol` label. If
|
|
491
|
+
repository policy prevents label creation or assignment, the bot stops rather
|
|
492
|
+
than publish work whose ownership cannot be verified. Existing work is trusted
|
|
493
|
+
only when the label, metadata, deterministic branch, repository, base, and Git
|
|
494
|
+
lease all agree. For HTTPS remotes, including private repositories, the bot
|
|
495
|
+
passes `GITHUB_TOKEN` to temporary Git processes as a host-scoped authorization
|
|
496
|
+
header and never puts it in a remote URL.
|
|
497
|
+
|
|
498
|
+
Two operational notes. PRs created with `GITHUB_TOKEN` do not trigger
|
|
499
|
+
`pull_request` workflows; store a fine-grained PAT or GitHub App token as the
|
|
500
|
+
optional `MODEL_EOL_BOT_TOKEN` secret when checks must run. And the workflow
|
|
501
|
+
splits privileges: only the provider secrets you add for the eval live in the
|
|
502
|
+
read-only evaluation job, and write tokens live only in the reconciliation job.
|
|
397
503
|
|
|
398
504
|
## Keeping the feeds honest
|
|
399
505
|
|
|
@@ -403,68 +509,76 @@ node refresh/refresh.mjs --distributor aws-bedrock,vertex-ai # distributor lifec
|
|
|
403
509
|
node scripts/feed-changelog.mjs # local rendering of the hosted Atom feed
|
|
404
510
|
```
|
|
405
511
|
|
|
406
|
-
|
|
407
|
-
|
|
408
|
-
|
|
409
|
-
|
|
410
|
-
|
|
411
|
-
|
|
412
|
-
|
|
512
|
+
A parse failure stops the run. The refresh never writes a guessed feed.
|
|
513
|
+
|
|
514
|
+
This runs automatically. A weekly workflow
|
|
515
|
+
(`.github/workflows/feed-refresh.yml`, Mondays 05:23 UTC) checks the live
|
|
516
|
+
sources and opens a PR with the semantic diff, plus an updated README freshness
|
|
517
|
+
line, whenever something material changed. If an earlier refresh PR is still
|
|
518
|
+
open and no human has touched it, the run updates that PR instead of opening
|
|
519
|
+
another one.
|
|
520
|
+
|
|
521
|
+
The first live runs found real errors. They caught the hand-compiled feeds
|
|
522
|
+
drifting from Anthropic's recommendations within one week, and corrected a
|
|
523
|
+
hand-compiled Bedrock date that was three months wrong.
|
|
413
524
|
|
|
414
|
-
Every successful live refresh
|
|
525
|
+
Every successful live refresh produces a byte-exact receipt. The Pages workflow
|
|
415
526
|
publishes the [public contract](https://thossullivan.github.io/model-eol/) only
|
|
416
|
-
when that receipt
|
|
417
|
-
`last_checked` immediately
|
|
418
|
-
|
|
419
|
-
|
|
420
|
-
|
|
421
|
-
|
|
422
|
-
The refresh also
|
|
423
|
-
|
|
424
|
-
Flash
|
|
527
|
+
when that receipt matches the exact feeds on `main`. A no-change check advances
|
|
528
|
+
`last_checked` immediately. A material-change run waits for its feed PR to
|
|
529
|
+
merge. The feed's `generated` date changes only when source data changes. The
|
|
530
|
+
hosted contract contains canonical Draft-07 schemas, feeds with SHA-256
|
|
531
|
+
receipts, and the Atom changelog.
|
|
532
|
+
|
|
533
|
+
The refresh also removes dates. On August 3, Google removed the October 16
|
|
534
|
+
earliest shutdown dates it had listed for the Gemini 2.5 Pro, Flash, and
|
|
535
|
+
Flash-Lite GA models. The pipeline opened
|
|
425
536
|
[a PR with the semantic retraction](https://github.com/thossullivan/model-eol/pull/60),
|
|
426
|
-
and downstream checks stopped warning
|
|
537
|
+
and downstream checks stopped warning about dates Google no longer listed. A
|
|
427
538
|
lifecycle tracker has to retract stale alarms as well as add new ones.
|
|
428
539
|
|
|
429
540
|
## The actual ask
|
|
430
541
|
|
|
431
|
-
The
|
|
432
|
-
deprecation *pages* because customers asked. A `retirement_date` field
|
|
433
|
-
endpoint, or this feed at a well-known URL,
|
|
434
|
-
|
|
435
|
-
convergence
|
|
436
|
-
|
|
437
|
-
|
|
542
|
+
The goal is the format, not another checker. Providers already publish
|
|
543
|
+
deprecation *pages* because customers asked for them. A `retirement_date` field
|
|
544
|
+
in the models endpoint, or this feed at a well-known URL, would let every
|
|
545
|
+
existing tracker read the same data. endoflife.date now tracks Claude with
|
|
546
|
+
nearly this field set, so the convergence has started. If you maintain one of
|
|
547
|
+
the community trackers, agreeing on a shared schema, this one or a better one,
|
|
548
|
+
is the most useful thing any of us can do here.
|
|
438
549
|
|
|
439
|
-
## Status
|
|
550
|
+
## Status
|
|
440
551
|
|
|
441
|
-
- Feeds refresh automatically
|
|
442
|
-
repo
|
|
552
|
+
- Feeds refresh automatically. The weekly feed-refresh workflow is live in this
|
|
553
|
+
repo. Parse failures fail the run, and material changes become a reviewed PR.
|
|
443
554
|
CI runs the full suite on every push and PR. The copy-ready bot workflow ships
|
|
444
|
-
as `bot.yml.example` and
|
|
445
|
-
|
|
555
|
+
as `bot.yml.example` and uses the published package, not local copies of the
|
|
556
|
+
tooling.
|
|
446
557
|
- The [public contract](https://thossullivan.github.io/model-eol/) publishes
|
|
447
558
|
canonical schemas, feeds, refresh health, a publication manifest, and Atom.
|
|
448
559
|
Each deployment is bound to an exact feed-refresh receipt and verified against
|
|
449
560
|
every live asset. `model-eol validate` checks feeds, repository policy, check
|
|
450
561
|
reports, inventories, schedules, alerts, and plans against the same
|
|
451
562
|
zero-dependency runtime contracts.
|
|
452
|
-
- Current-model entries
|
|
453
|
-
refresh runs with provider API keys for the models endpoints.
|
|
454
|
-
|
|
455
|
-
|
|
456
|
-
|
|
457
|
-
|
|
458
|
-
|
|
459
|
-
|
|
460
|
-
|
|
461
|
-
|
|
462
|
-
|
|
463
|
-
|
|
563
|
+
- Current-model entries, and therefore policy-floor horizons, populate only when
|
|
564
|
+
the refresh runs with provider API keys for the models endpoints. Anthropic's
|
|
565
|
+
tentative dates come from its public deprecations page and need no key.
|
|
566
|
+
- The checker matches known IDs only. It does not discover models that are
|
|
567
|
+
absent from the feeds. This is deliberate: a CI gate needs precision more than
|
|
568
|
+
discovery.
|
|
569
|
+
- Fetchers: OpenAI, Anthropic, Google, aws-bedrock, and vertex-ai are live.
|
|
570
|
+
Azure dates are carried where OpenAI's own page publishes them. A standalone
|
|
571
|
+
Azure lifecycle fetcher is not built yet.
|
|
572
|
+
- On npm as [`model-eol`](https://www.npmjs.com/package/model-eol). Material
|
|
573
|
+
feed changes republish automatically as patch versions, with trusted
|
|
574
|
+
publishing and provenance. Code releases require an explicit stable version.
|
|
575
|
+
`npx model-eol` therefore always checks against current dates.
|
|
576
|
+
- Not yet: feed signing, and authenticated account-level gateway or cloud
|
|
577
|
+
resolvers.
|
|
464
578
|
|
|
465
579
|
## Contributing
|
|
466
580
|
|
|
467
|
-
The most valuable contribution is a feed correction with a source URL
|
|
468
|
-
|
|
469
|
-
[CONTRIBUTING.md](CONTRIBUTING.md) for that workflow, the zero-dependency
|
|
470
|
-
|
|
581
|
+
The most valuable contribution is a feed correction with a source URL. The one
|
|
582
|
+
rule is that `feeds/*.json` are generated, never hand-edited. See
|
|
583
|
+
[CONTRIBUTING.md](CONTRIBUTING.md) for that workflow, the zero-dependency rule,
|
|
584
|
+
and what a new publisher parser needs.
|