@llmintel/cli 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +118 -0
- package/action.yml +61 -0
- package/dist/index.js +806 -0
- package/package.json +53 -0
package/README.md
ADDED
|
@@ -0,0 +1,118 @@
|
|
|
1
|
+
# @llmintel/cli
|
|
2
|
+
|
|
3
|
+
Fail your CI build when it references **retired or soon-to-be-retired AI models**.
|
|
4
|
+
|
|
5
|
+
`llmintel check` scans your code/config for referenced model ids (and/or takes explicit ids),
|
|
6
|
+
looks up their lifecycle state via the [LLMIntel API](https://llmintel.vercel.app/docs), and exits
|
|
7
|
+
non-zero when any referenced model is **retired or past its retirement date** — so you migrate on
|
|
8
|
+
your schedule, not the provider's shutoff date.
|
|
9
|
+
|
|
10
|
+
## Install
|
|
11
|
+
|
|
12
|
+
No install required — run it with `npx`:
|
|
13
|
+
|
|
14
|
+
```bash
|
|
15
|
+
npx @llmintel/cli@latest check src config
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
Or add it as a dev dependency:
|
|
19
|
+
|
|
20
|
+
```bash
|
|
21
|
+
npm install --save-dev @llmintel/cli
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
## Authenticate
|
|
25
|
+
|
|
26
|
+
Create an API key from your [dashboard](https://llmintel.vercel.app/dashboard) and expose it as an
|
|
27
|
+
environment variable:
|
|
28
|
+
|
|
29
|
+
```bash
|
|
30
|
+
export LLMINTEL_API_KEY="mc_live_..."
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
## Usage
|
|
34
|
+
|
|
35
|
+
```bash
|
|
36
|
+
# Scan files/directories for referenced model ids and gate on lifecycle state:
|
|
37
|
+
llmintel check src config
|
|
38
|
+
|
|
39
|
+
# Check explicit ids/aliases without scanning:
|
|
40
|
+
llmintel check --models "gpt-4o,claude-3-opus-20240229"
|
|
41
|
+
|
|
42
|
+
# Use a config file:
|
|
43
|
+
llmintel check --config llmintel.json
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
### Options
|
|
47
|
+
|
|
48
|
+
| Flag | Description |
|
|
49
|
+
| --------------------- | ---------------------------------------------------------------------- |
|
|
50
|
+
| `-m, --models <a,b,c>` | Explicit model ids/aliases to check (repeatable) |
|
|
51
|
+
| `-c, --config <file>` | JSON config: `{ "models": [...], "paths": [...], "warnDays": 90 }` |
|
|
52
|
+
| `--api-url <url>` | API base URL (env `LLMINTEL_API_URL`, default `https://llmintel.vercel.app`) |
|
|
53
|
+
| `--api-key <key>` | API key (env `LLMINTEL_API_KEY`) |
|
|
54
|
+
| `--warn-days <n>` | Warn when a model retires within this many days (default `90`) |
|
|
55
|
+
| `--fail-on-warn` | Treat warnings (deprecated / retiring soon) as build failures |
|
|
56
|
+
| `--fail-on-unknown` | Treat references LLMIntel does not track as build failures |
|
|
57
|
+
| `--no-policy` | Ignore your account's central policy; use only the flags above |
|
|
58
|
+
| `--optimize` | Also print advisory cheaper/faster model suggestions (paid; never fails the build) |
|
|
59
|
+
| `--json` | Machine-readable JSON output |
|
|
60
|
+
| `-q, --quiet` | Only print failures |
|
|
61
|
+
|
|
62
|
+
## Central gate policy
|
|
63
|
+
|
|
64
|
+
By default `check` fails only on **retired** (or past-due) models — free forever. On a paid plan
|
|
65
|
+
you can configure a stricter policy **once** in your [dashboard](https://llmintel.vercel.app/dashboard)
|
|
66
|
+
and every pipeline picks it up automatically: fail on `deprecated`/`retiring`, set the warn
|
|
67
|
+
window, and fail on untracked references.
|
|
68
|
+
|
|
69
|
+
`check` fetches this policy on each run and uses it as the baseline. Explicit local flags
|
|
70
|
+
(`--warn-days`, `--fail-on-unknown`) override it; pass `--no-policy` to ignore it entirely. Free
|
|
71
|
+
accounts are always clamped to the retired-only gate regardless of any saved policy.
|
|
72
|
+
|
|
73
|
+
## Auto-watch your codebase (`sync`)
|
|
74
|
+
|
|
75
|
+
`llmintel sync` discovers the models your code references — the same scan as `check` — and
|
|
76
|
+
registers them as your account's **watched set** server-side. Combined with push alerts
|
|
77
|
+
(webhook / Slack / PagerDuty), this gives you "fire and forget" coverage: run `sync` in CI on
|
|
78
|
+
merge to `main`, and you'll be notified the moment a model you actually use is deprecated or
|
|
79
|
+
retired, with no watchlist to curate by hand.
|
|
80
|
+
|
|
81
|
+
```bash
|
|
82
|
+
# Replace your watched set with whatever your code references now:
|
|
83
|
+
llmintel sync src config
|
|
84
|
+
|
|
85
|
+
# Preview the diff without writing anything:
|
|
86
|
+
llmintel sync src config --dry-run
|
|
87
|
+
|
|
88
|
+
# Only add models; never remove ones you watch elsewhere:
|
|
89
|
+
llmintel sync src config --no-prune
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
`sync` requires a paid plan with auto-watch enabled and an API key linked to an account. It prints
|
|
93
|
+
the diff of `added` / `removed` / `unchanged` watches plus any `unresolved` references.
|
|
94
|
+
|
|
95
|
+
### Exit codes
|
|
96
|
+
|
|
97
|
+
| Code | Meaning |
|
|
98
|
+
| ---- | ------------------------------------------------------------- |
|
|
99
|
+
| `0` | All referenced models healthy |
|
|
100
|
+
| `1` | One or more errors (retired / past due), or warnings with `--fail-on-warn` |
|
|
101
|
+
| `2` | Usage / configuration error |
|
|
102
|
+
| `3` | API or network error |
|
|
103
|
+
|
|
104
|
+
## GitHub Actions
|
|
105
|
+
|
|
106
|
+
A composite action ships alongside the CLI:
|
|
107
|
+
|
|
108
|
+
```yaml
|
|
109
|
+
- uses: hivemindunit/llmintel-cli/packages/cli@main
|
|
110
|
+
with:
|
|
111
|
+
api-key: ${{ secrets.LLMINTEL_API_KEY }}
|
|
112
|
+
paths: "src config"
|
|
113
|
+
warn-days: "90"
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
## License
|
|
117
|
+
|
|
118
|
+
MIT
|
package/action.yml
ADDED
|
@@ -0,0 +1,61 @@
|
|
|
1
|
+
name: "LLMIntel Check"
|
|
2
|
+
description: "Fail your build when it references retired or soon-to-be-retired AI models."
|
|
3
|
+
author: "LLMIntel"
|
|
4
|
+
branding:
|
|
5
|
+
icon: "clock"
|
|
6
|
+
color: "orange"
|
|
7
|
+
|
|
8
|
+
inputs:
|
|
9
|
+
api-key:
|
|
10
|
+
description: "LLMIntel API key. Store it as a repository secret and pass it here."
|
|
11
|
+
required: true
|
|
12
|
+
paths:
|
|
13
|
+
description: "Space-separated files/directories to scan for model references."
|
|
14
|
+
required: false
|
|
15
|
+
default: "."
|
|
16
|
+
models:
|
|
17
|
+
description: "Comma-separated explicit model ids/aliases to check (in addition to scanned paths)."
|
|
18
|
+
required: false
|
|
19
|
+
default: ""
|
|
20
|
+
config:
|
|
21
|
+
description: "Path to a JSON config file ({ models, paths, warnDays, apiUrl })."
|
|
22
|
+
required: false
|
|
23
|
+
default: ""
|
|
24
|
+
api-url:
|
|
25
|
+
description: "LLMIntel API base URL."
|
|
26
|
+
required: false
|
|
27
|
+
default: "https://llmintel.vercel.app"
|
|
28
|
+
warn-days:
|
|
29
|
+
description: "Warn when a model retires within this many days."
|
|
30
|
+
required: false
|
|
31
|
+
default: "90"
|
|
32
|
+
fail-on-warn:
|
|
33
|
+
description: "Treat warnings (deprecated / retiring soon) as build failures."
|
|
34
|
+
required: false
|
|
35
|
+
default: "false"
|
|
36
|
+
fail-on-unknown:
|
|
37
|
+
description: "Treat references that LLMIntel does not track as build failures."
|
|
38
|
+
required: false
|
|
39
|
+
default: "false"
|
|
40
|
+
version:
|
|
41
|
+
description: "npm version/tag of the CLI to run (e.g. latest, 0.1.0)."
|
|
42
|
+
required: false
|
|
43
|
+
default: "latest"
|
|
44
|
+
|
|
45
|
+
runs:
|
|
46
|
+
using: "composite"
|
|
47
|
+
steps:
|
|
48
|
+
- name: Run llmintel check
|
|
49
|
+
shell: bash
|
|
50
|
+
env:
|
|
51
|
+
LLMINTEL_API_KEY: ${{ inputs.api-key }}
|
|
52
|
+
LLMINTEL_API_URL: ${{ inputs.api-url }}
|
|
53
|
+
run: |
|
|
54
|
+
args=(check)
|
|
55
|
+
for p in ${{ inputs.paths }}; do args+=("$p"); done
|
|
56
|
+
if [ -n "${{ inputs.models }}" ]; then args+=(--models "${{ inputs.models }}"); fi
|
|
57
|
+
if [ -n "${{ inputs.config }}" ]; then args+=(--config "${{ inputs.config }}"); fi
|
|
58
|
+
args+=(--warn-days "${{ inputs.warn-days }}")
|
|
59
|
+
if [ "${{ inputs.fail-on-warn }}" = "true" ]; then args+=(--fail-on-warn); fi
|
|
60
|
+
if [ "${{ inputs.fail-on-unknown }}" = "true" ]; then args+=(--fail-on-unknown); fi
|
|
61
|
+
npx --yes @llmintel/cli@${{ inputs.version }} "${args[@]}"
|
package/dist/index.js
ADDED
|
@@ -0,0 +1,806 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
|
|
3
|
+
// src/client.ts
|
|
4
|
+
var ApiError = class extends Error {
|
|
5
|
+
constructor(message, status, code) {
|
|
6
|
+
super(message);
|
|
7
|
+
this.status = status;
|
|
8
|
+
this.code = code;
|
|
9
|
+
this.name = "ApiError";
|
|
10
|
+
}
|
|
11
|
+
status;
|
|
12
|
+
code;
|
|
13
|
+
};
|
|
14
|
+
function networkError(base, cause) {
|
|
15
|
+
const raw = cause instanceof Error ? cause.message : String(cause);
|
|
16
|
+
const text = `${raw} ${String(cause?.cause?.code ?? "")}`;
|
|
17
|
+
const unreachable = /ENOTFOUND|EAI_AGAIN|getaddrinfo|ECONNREFUSED|ECONNRESET|ETIMEDOUT|UND_ERR/i.test(text);
|
|
18
|
+
if (unreachable) {
|
|
19
|
+
return new ApiError(
|
|
20
|
+
`Could not reach the LLMIntel API at ${base} (${raw}). Check the host is correct \u2014 override it with --api-url or the LLMINTEL_API_URL env var.`
|
|
21
|
+
);
|
|
22
|
+
}
|
|
23
|
+
return new ApiError(`Network error contacting ${base}: ${raw}`);
|
|
24
|
+
}
|
|
25
|
+
var PAGE_SIZE = 500;
|
|
26
|
+
async function fetchAllModels(options) {
|
|
27
|
+
const fetchImpl = options.fetchImpl ?? fetch;
|
|
28
|
+
const base = options.baseUrl.replace(/\/$/, "");
|
|
29
|
+
const all = [];
|
|
30
|
+
let offset = 0;
|
|
31
|
+
for (; ; ) {
|
|
32
|
+
const url = `${base}/v1/models?limit=${PAGE_SIZE}&offset=${offset}`;
|
|
33
|
+
let response;
|
|
34
|
+
try {
|
|
35
|
+
response = await fetchImpl(url, {
|
|
36
|
+
headers: { authorization: `Bearer ${options.apiKey}`, accept: "application/json" }
|
|
37
|
+
});
|
|
38
|
+
} catch (cause) {
|
|
39
|
+
throw networkError(base, cause);
|
|
40
|
+
}
|
|
41
|
+
if (!response.ok) {
|
|
42
|
+
let code;
|
|
43
|
+
let message = `${response.status} ${response.statusText}`;
|
|
44
|
+
try {
|
|
45
|
+
const body2 = await response.json();
|
|
46
|
+
code = body2.error;
|
|
47
|
+
if (body2.message) message = body2.message;
|
|
48
|
+
} catch {
|
|
49
|
+
}
|
|
50
|
+
throw new ApiError(message, response.status, code);
|
|
51
|
+
}
|
|
52
|
+
const body = await response.json();
|
|
53
|
+
all.push(...body.data);
|
|
54
|
+
if (body.data.length < PAGE_SIZE) break;
|
|
55
|
+
offset += PAGE_SIZE;
|
|
56
|
+
}
|
|
57
|
+
return all;
|
|
58
|
+
}
|
|
59
|
+
async function syncWatches(options) {
|
|
60
|
+
const fetchImpl = options.fetchImpl ?? fetch;
|
|
61
|
+
const base = options.baseUrl.replace(/\/$/, "");
|
|
62
|
+
const url = `${base}/v1/watches`;
|
|
63
|
+
let response;
|
|
64
|
+
try {
|
|
65
|
+
response = await fetchImpl(url, {
|
|
66
|
+
method: "PUT",
|
|
67
|
+
headers: {
|
|
68
|
+
authorization: `Bearer ${options.apiKey}`,
|
|
69
|
+
accept: "application/json",
|
|
70
|
+
"content-type": "application/json"
|
|
71
|
+
},
|
|
72
|
+
body: JSON.stringify({ models: options.models, prune: options.prune })
|
|
73
|
+
});
|
|
74
|
+
} catch (cause) {
|
|
75
|
+
throw networkError(base, cause);
|
|
76
|
+
}
|
|
77
|
+
if (!response.ok) {
|
|
78
|
+
let code;
|
|
79
|
+
let message = `${response.status} ${response.statusText}`;
|
|
80
|
+
try {
|
|
81
|
+
const errBody = await response.json();
|
|
82
|
+
code = errBody.error;
|
|
83
|
+
if (errBody.message) message = errBody.message;
|
|
84
|
+
} catch {
|
|
85
|
+
}
|
|
86
|
+
throw new ApiError(message, response.status, code);
|
|
87
|
+
}
|
|
88
|
+
const body = await response.json();
|
|
89
|
+
return body.data;
|
|
90
|
+
}
|
|
91
|
+
async function fetchPolicy(options) {
|
|
92
|
+
const fetchImpl = options.fetchImpl ?? fetch;
|
|
93
|
+
const base = options.baseUrl.replace(/\/$/, "");
|
|
94
|
+
const url = `${base}/v1/policy`;
|
|
95
|
+
let response;
|
|
96
|
+
try {
|
|
97
|
+
response = await fetchImpl(url, {
|
|
98
|
+
headers: { authorization: `Bearer ${options.apiKey}`, accept: "application/json" }
|
|
99
|
+
});
|
|
100
|
+
} catch (cause) {
|
|
101
|
+
throw networkError(base, cause);
|
|
102
|
+
}
|
|
103
|
+
if (!response.ok) {
|
|
104
|
+
let code;
|
|
105
|
+
let message = `${response.status} ${response.statusText}`;
|
|
106
|
+
try {
|
|
107
|
+
const errBody = await response.json();
|
|
108
|
+
code = errBody.error;
|
|
109
|
+
if (errBody.message) message = errBody.message;
|
|
110
|
+
} catch {
|
|
111
|
+
}
|
|
112
|
+
throw new ApiError(message, response.status, code);
|
|
113
|
+
}
|
|
114
|
+
const body = await response.json();
|
|
115
|
+
return body.data;
|
|
116
|
+
}
|
|
117
|
+
async function fetchOptimization(options) {
|
|
118
|
+
const fetchImpl = options.fetchImpl ?? fetch;
|
|
119
|
+
const base = options.baseUrl.replace(/\/$/, "");
|
|
120
|
+
const path = options.modelId.split("/").map(encodeURIComponent).join("/");
|
|
121
|
+
const url = `${base}/v1/models/${path}`;
|
|
122
|
+
let response;
|
|
123
|
+
try {
|
|
124
|
+
response = await fetchImpl(url, {
|
|
125
|
+
headers: { authorization: `Bearer ${options.apiKey}`, accept: "application/json" }
|
|
126
|
+
});
|
|
127
|
+
} catch (cause) {
|
|
128
|
+
throw networkError(base, cause);
|
|
129
|
+
}
|
|
130
|
+
if (!response.ok) {
|
|
131
|
+
let code;
|
|
132
|
+
let message = `${response.status} ${response.statusText}`;
|
|
133
|
+
try {
|
|
134
|
+
const errBody = await response.json();
|
|
135
|
+
code = errBody.error;
|
|
136
|
+
if (errBody.message) message = errBody.message;
|
|
137
|
+
} catch {
|
|
138
|
+
}
|
|
139
|
+
throw new ApiError(message, response.status, code);
|
|
140
|
+
}
|
|
141
|
+
const body = await response.json();
|
|
142
|
+
return body.data.optimization ?? null;
|
|
143
|
+
}
|
|
144
|
+
|
|
145
|
+
// src/config.ts
|
|
146
|
+
import { readFile } from "fs/promises";
|
|
147
|
+
var ConfigError = class extends Error {
|
|
148
|
+
};
|
|
149
|
+
async function loadConfig(path) {
|
|
150
|
+
let raw;
|
|
151
|
+
try {
|
|
152
|
+
raw = await readFile(path, "utf8");
|
|
153
|
+
} catch (cause) {
|
|
154
|
+
throw new ConfigError(`Cannot read config file "${path}": ${cause.message}`);
|
|
155
|
+
}
|
|
156
|
+
let parsed;
|
|
157
|
+
try {
|
|
158
|
+
parsed = JSON.parse(raw);
|
|
159
|
+
} catch (cause) {
|
|
160
|
+
throw new ConfigError(`Invalid JSON in "${path}": ${cause.message}`);
|
|
161
|
+
}
|
|
162
|
+
if (typeof parsed !== "object" || parsed === null) {
|
|
163
|
+
throw new ConfigError(`Config "${path}" must be a JSON object.`);
|
|
164
|
+
}
|
|
165
|
+
const config = parsed;
|
|
166
|
+
const result = {};
|
|
167
|
+
if (config.models !== void 0) {
|
|
168
|
+
if (!Array.isArray(config.models) || config.models.some((m) => typeof m !== "string")) {
|
|
169
|
+
throw new ConfigError(`Config "models" must be an array of strings.`);
|
|
170
|
+
}
|
|
171
|
+
result.models = config.models;
|
|
172
|
+
}
|
|
173
|
+
if (config.paths !== void 0) {
|
|
174
|
+
if (!Array.isArray(config.paths) || config.paths.some((p) => typeof p !== "string")) {
|
|
175
|
+
throw new ConfigError(`Config "paths" must be an array of strings.`);
|
|
176
|
+
}
|
|
177
|
+
result.paths = config.paths;
|
|
178
|
+
}
|
|
179
|
+
if (config.warnDays !== void 0) {
|
|
180
|
+
if (typeof config.warnDays !== "number" || config.warnDays < 0) {
|
|
181
|
+
throw new ConfigError(`Config "warnDays" must be a non-negative number.`);
|
|
182
|
+
}
|
|
183
|
+
result.warnDays = Math.floor(config.warnDays);
|
|
184
|
+
}
|
|
185
|
+
if (config.apiUrl !== void 0) {
|
|
186
|
+
if (typeof config.apiUrl !== "string") throw new ConfigError(`Config "apiUrl" must be a string.`);
|
|
187
|
+
result.apiUrl = config.apiUrl;
|
|
188
|
+
}
|
|
189
|
+
return result;
|
|
190
|
+
}
|
|
191
|
+
|
|
192
|
+
// src/extract.ts
|
|
193
|
+
var QUOTED = /['"`]([A-Za-z0-9][A-Za-z0-9._/-]{2,80})['"`]/g;
|
|
194
|
+
function looksLikeModel(token) {
|
|
195
|
+
if (!/\d/.test(token)) return false;
|
|
196
|
+
if (!/[-/]/.test(token)) return false;
|
|
197
|
+
if (token.includes("://") || token.startsWith("/") || token.includes("\\")) return false;
|
|
198
|
+
if (/\.(js|ts|tsx|jsx|json|py|go|rb|java|css|html|md|png|svg|lock)$/i.test(token)) return false;
|
|
199
|
+
return true;
|
|
200
|
+
}
|
|
201
|
+
function extractReferences(text, source) {
|
|
202
|
+
const found = /* @__PURE__ */ new Map();
|
|
203
|
+
for (const match of text.matchAll(QUOTED)) {
|
|
204
|
+
const token = match[1];
|
|
205
|
+
if (token && looksLikeModel(token) && !found.has(token)) {
|
|
206
|
+
found.set(token, { value: token, source });
|
|
207
|
+
}
|
|
208
|
+
}
|
|
209
|
+
return [...found.values()];
|
|
210
|
+
}
|
|
211
|
+
function dedupeReferences(lists) {
|
|
212
|
+
const seen = /* @__PURE__ */ new Map();
|
|
213
|
+
for (const list of lists) {
|
|
214
|
+
for (const ref of list) {
|
|
215
|
+
if (!seen.has(ref.value)) seen.set(ref.value, ref);
|
|
216
|
+
}
|
|
217
|
+
}
|
|
218
|
+
return [...seen.values()];
|
|
219
|
+
}
|
|
220
|
+
|
|
221
|
+
// src/gate.ts
|
|
222
|
+
var DAY_MS = 24 * 60 * 60 * 1e3;
|
|
223
|
+
function daysUntil(isoDate, now) {
|
|
224
|
+
const target = (/* @__PURE__ */ new Date(`${isoDate}T00:00:00Z`)).getTime();
|
|
225
|
+
const today = Date.UTC(now.getUTCFullYear(), now.getUTCMonth(), now.getUTCDate());
|
|
226
|
+
return Math.round((target - today) / DAY_MS);
|
|
227
|
+
}
|
|
228
|
+
function buildIndex(models) {
|
|
229
|
+
const index = /* @__PURE__ */ new Map();
|
|
230
|
+
for (const model of models) {
|
|
231
|
+
index.set(model.id.toLowerCase(), model);
|
|
232
|
+
for (const alias of model.aliases) {
|
|
233
|
+
index.set(alias.toLowerCase(), model);
|
|
234
|
+
}
|
|
235
|
+
}
|
|
236
|
+
return index;
|
|
237
|
+
}
|
|
238
|
+
function evaluateReference(reference, index, options, now) {
|
|
239
|
+
const model = index.get(reference.value.toLowerCase()) ?? null;
|
|
240
|
+
const failOn = options.failOn ?? ["retired"];
|
|
241
|
+
if (!model) {
|
|
242
|
+
return {
|
|
243
|
+
reference,
|
|
244
|
+
model: null,
|
|
245
|
+
severity: options.failOnUnknown ? "error" : "unknown",
|
|
246
|
+
reason: "not tracked by LLMIntel",
|
|
247
|
+
daysUntilRetirement: null
|
|
248
|
+
};
|
|
249
|
+
}
|
|
250
|
+
const days = model.retirementDate ? daysUntil(model.retirementDate, now) : null;
|
|
251
|
+
if (failOn.includes(model.lifecycleState)) {
|
|
252
|
+
const reason = model.lifecycleState === "retired" ? "retired \u2014 API calls fail" : `${model.lifecycleState} (policy fails on this state)`;
|
|
253
|
+
return { reference, model, severity: "error", reason, daysUntilRetirement: days };
|
|
254
|
+
}
|
|
255
|
+
if (days !== null && days < 0) {
|
|
256
|
+
return {
|
|
257
|
+
reference,
|
|
258
|
+
model,
|
|
259
|
+
severity: "error",
|
|
260
|
+
reason: `retirement date ${model.retirementDate} has passed`,
|
|
261
|
+
daysUntilRetirement: days
|
|
262
|
+
};
|
|
263
|
+
}
|
|
264
|
+
const within = days !== null && days <= options.warnDays;
|
|
265
|
+
if (model.lifecycleState === "deprecated" || model.lifecycleState === "retiring" || within) {
|
|
266
|
+
const when = days !== null ? `retires in ${days} day${days === 1 ? "" : "s"} (${model.retirementDate})` : model.lifecycleState;
|
|
267
|
+
return { reference, model, severity: "warn", reason: when, daysUntilRetirement: days };
|
|
268
|
+
}
|
|
269
|
+
return { reference, model, severity: "ok", reason: model.lifecycleState, daysUntilRetirement: days };
|
|
270
|
+
}
|
|
271
|
+
var EMPTY_COUNTS = () => ({ ok: 0, warn: 0, error: 0, unknown: 0 });
|
|
272
|
+
function buildReport(references, models, options, now = /* @__PURE__ */ new Date()) {
|
|
273
|
+
const index = buildIndex(models);
|
|
274
|
+
const findings = references.map((ref) => evaluateReference(ref, index, options, now));
|
|
275
|
+
const counts = EMPTY_COUNTS();
|
|
276
|
+
for (const finding of findings) counts[finding.severity] += 1;
|
|
277
|
+
let exitCode = 0;
|
|
278
|
+
if (counts.error > 0) exitCode = 1;
|
|
279
|
+
else if (options.failOnWarn && counts.warn > 0) exitCode = 1;
|
|
280
|
+
return { findings, counts, exitCode };
|
|
281
|
+
}
|
|
282
|
+
|
|
283
|
+
// src/report.ts
|
|
284
|
+
var SYMBOLS = {
|
|
285
|
+
ok: "ok ",
|
|
286
|
+
warn: "WARN ",
|
|
287
|
+
error: "FAIL ",
|
|
288
|
+
unknown: "? "
|
|
289
|
+
};
|
|
290
|
+
function formatHuman(report, quiet) {
|
|
291
|
+
const lines = [];
|
|
292
|
+
const rows = report.findings.filter((f) => quiet ? f.severity === "error" || f.severity === "warn" : true);
|
|
293
|
+
for (const finding of rows) {
|
|
294
|
+
const id = finding.model?.id ?? finding.reference.value;
|
|
295
|
+
lines.push(` ${SYMBOLS[finding.severity]} ${id} \u2014 ${finding.reason} [${finding.reference.source}]`);
|
|
296
|
+
}
|
|
297
|
+
if (lines.length > 0) lines.push("");
|
|
298
|
+
lines.push(
|
|
299
|
+
`Checked ${report.findings.length} reference(s): ${report.counts.error} error, ${report.counts.warn} warn, ${report.counts.unknown} unknown, ${report.counts.ok} ok.`
|
|
300
|
+
);
|
|
301
|
+
return lines.join("\n");
|
|
302
|
+
}
|
|
303
|
+
function formatJson(report, optimizations = []) {
|
|
304
|
+
return JSON.stringify(
|
|
305
|
+
{
|
|
306
|
+
exitCode: report.exitCode,
|
|
307
|
+
counts: report.counts,
|
|
308
|
+
findings: report.findings.map((f) => ({
|
|
309
|
+
reference: f.reference.value,
|
|
310
|
+
source: f.reference.source,
|
|
311
|
+
modelId: f.model?.id ?? null,
|
|
312
|
+
provider: f.model?.provider ?? null,
|
|
313
|
+
lifecycleState: f.model?.lifecycleState ?? null,
|
|
314
|
+
retirementDate: f.model?.retirementDate ?? null,
|
|
315
|
+
daysUntilRetirement: f.daysUntilRetirement,
|
|
316
|
+
severity: f.severity,
|
|
317
|
+
reason: f.reason
|
|
318
|
+
})),
|
|
319
|
+
optimizations: optimizations.map((o) => ({
|
|
320
|
+
modelId: o.modelId,
|
|
321
|
+
candidateId: o.candidateId,
|
|
322
|
+
candidateProvider: o.candidateProvider,
|
|
323
|
+
reasons: o.reasons,
|
|
324
|
+
crossProvider: o.crossProvider
|
|
325
|
+
}))
|
|
326
|
+
},
|
|
327
|
+
null,
|
|
328
|
+
2
|
|
329
|
+
);
|
|
330
|
+
}
|
|
331
|
+
function formatOptimization(optimizations) {
|
|
332
|
+
if (optimizations.length === 0) return "";
|
|
333
|
+
const lines = ["", "Optimization suggestions (advisory \u2014 does not affect the build):"];
|
|
334
|
+
for (const o of optimizations) {
|
|
335
|
+
const switchNote = o.crossProvider ? ` (switch to ${o.candidateProvider})` : "";
|
|
336
|
+
lines.push(` ~ ${o.modelId} \u2192 ${o.candidateId}${switchNote}: ${o.reasons.join(", ")}`);
|
|
337
|
+
}
|
|
338
|
+
return lines.join("\n");
|
|
339
|
+
}
|
|
340
|
+
|
|
341
|
+
// src/scan.ts
|
|
342
|
+
import { readdir, readFile as readFile2, stat } from "fs/promises";
|
|
343
|
+
import { join } from "path";
|
|
344
|
+
var IGNORE_DIRS = /* @__PURE__ */ new Set([
|
|
345
|
+
"node_modules",
|
|
346
|
+
".git",
|
|
347
|
+
".next",
|
|
348
|
+
"dist",
|
|
349
|
+
"build",
|
|
350
|
+
"coverage",
|
|
351
|
+
".turbo",
|
|
352
|
+
".vercel",
|
|
353
|
+
"vendor",
|
|
354
|
+
"__pycache__"
|
|
355
|
+
]);
|
|
356
|
+
var MAX_FILE_BYTES = 2 * 1024 * 1024;
|
|
357
|
+
async function scanPath(path) {
|
|
358
|
+
let info;
|
|
359
|
+
try {
|
|
360
|
+
info = await stat(path);
|
|
361
|
+
} catch {
|
|
362
|
+
return [];
|
|
363
|
+
}
|
|
364
|
+
if (info.isDirectory()) {
|
|
365
|
+
const entries = await readdir(path, { withFileTypes: true });
|
|
366
|
+
const results = await Promise.all(
|
|
367
|
+
entries.map((entry) => {
|
|
368
|
+
if (entry.isDirectory() && IGNORE_DIRS.has(entry.name)) return Promise.resolve([]);
|
|
369
|
+
return scanPath(join(path, entry.name));
|
|
370
|
+
})
|
|
371
|
+
);
|
|
372
|
+
return results.flat();
|
|
373
|
+
}
|
|
374
|
+
if (info.size > MAX_FILE_BYTES) return [];
|
|
375
|
+
try {
|
|
376
|
+
const text = await readFile2(path, "utf8");
|
|
377
|
+
return extractReferences(text, path);
|
|
378
|
+
} catch {
|
|
379
|
+
return [];
|
|
380
|
+
}
|
|
381
|
+
}
|
|
382
|
+
async function scanPaths(paths) {
|
|
383
|
+
const lists = await Promise.all(paths.map(scanPath));
|
|
384
|
+
return lists.flat();
|
|
385
|
+
}
|
|
386
|
+
|
|
387
|
+
// src/sync.ts
|
|
388
|
+
function formatSyncHuman(summary, quiet) {
|
|
389
|
+
const lines = [];
|
|
390
|
+
const prefix = summary.dryRun ? "Would" : "Did";
|
|
391
|
+
if (!quiet) {
|
|
392
|
+
for (const id of summary.added) lines.push(` + ${id}`);
|
|
393
|
+
for (const id of summary.removed) lines.push(` - ${id}`);
|
|
394
|
+
}
|
|
395
|
+
for (const ref of summary.unresolved) lines.push(` ? ${ref} \u2014 not tracked by LLMIntel`);
|
|
396
|
+
if (lines.length > 0) lines.push("");
|
|
397
|
+
lines.push(
|
|
398
|
+
`${prefix} sync ${summary.discovered} discovered reference(s): ${summary.added.length} added, ${summary.removed.length} removed, ${summary.unchanged.length} unchanged, ${summary.unresolved.length} unresolved.`
|
|
399
|
+
);
|
|
400
|
+
return lines.join("\n");
|
|
401
|
+
}
|
|
402
|
+
function formatSyncJson(summary) {
|
|
403
|
+
return JSON.stringify(
|
|
404
|
+
{
|
|
405
|
+
dryRun: summary.dryRun,
|
|
406
|
+
discovered: summary.discovered,
|
|
407
|
+
added: summary.added,
|
|
408
|
+
removed: summary.removed,
|
|
409
|
+
unchanged: summary.unchanged,
|
|
410
|
+
unresolved: summary.unresolved
|
|
411
|
+
},
|
|
412
|
+
null,
|
|
413
|
+
2
|
|
414
|
+
);
|
|
415
|
+
}
|
|
416
|
+
|
|
417
|
+
// src/args.ts
|
|
418
|
+
var DEFAULT_API_URL = "https://llmintel.vercel.app";
|
|
419
|
+
var DEFAULT_WARN_DAYS = 90;
|
|
420
|
+
var UsageError = class extends Error {
|
|
421
|
+
};
|
|
422
|
+
function parseModelsList(value) {
|
|
423
|
+
return value.split(",").map((s) => s.trim()).filter(Boolean);
|
|
424
|
+
}
|
|
425
|
+
function parseArgs(argv, env = process.env) {
|
|
426
|
+
const options = {
|
|
427
|
+
command: "help",
|
|
428
|
+
models: [],
|
|
429
|
+
paths: [],
|
|
430
|
+
config: null,
|
|
431
|
+
apiUrl: env.LLMINTEL_API_URL ?? DEFAULT_API_URL,
|
|
432
|
+
apiKey: env.LLMINTEL_API_KEY ?? null,
|
|
433
|
+
warnDays: DEFAULT_WARN_DAYS,
|
|
434
|
+
failOnWarn: false,
|
|
435
|
+
failOnUnknown: false,
|
|
436
|
+
json: false,
|
|
437
|
+
quiet: false,
|
|
438
|
+
dryRun: false,
|
|
439
|
+
prune: true,
|
|
440
|
+
usePolicy: true,
|
|
441
|
+
optimize: false,
|
|
442
|
+
explicit: { warnDays: false, failOnWarn: false, failOnUnknown: false }
|
|
443
|
+
};
|
|
444
|
+
const positional = [];
|
|
445
|
+
let i = 0;
|
|
446
|
+
const first = argv[0];
|
|
447
|
+
if (first === "check" || first === "sync" || first === "help" || first === "version") {
|
|
448
|
+
options.command = first;
|
|
449
|
+
i = 1;
|
|
450
|
+
} else if (first === "--version" || first === "-v") {
|
|
451
|
+
options.command = "version";
|
|
452
|
+
return options;
|
|
453
|
+
} else if (first === "--help" || first === "-h" || first === void 0) {
|
|
454
|
+
options.command = "help";
|
|
455
|
+
return options;
|
|
456
|
+
} else {
|
|
457
|
+
options.command = "check";
|
|
458
|
+
}
|
|
459
|
+
const need = (flag, value) => {
|
|
460
|
+
if (value === void 0) throw new UsageError(`Missing value for ${flag}`);
|
|
461
|
+
return value;
|
|
462
|
+
};
|
|
463
|
+
for (; i < argv.length; i++) {
|
|
464
|
+
const arg = argv[i];
|
|
465
|
+
if (arg === void 0) continue;
|
|
466
|
+
switch (arg) {
|
|
467
|
+
case "--models":
|
|
468
|
+
case "-m":
|
|
469
|
+
options.models.push(...parseModelsList(need(arg, argv[++i])));
|
|
470
|
+
break;
|
|
471
|
+
case "--config":
|
|
472
|
+
case "-c":
|
|
473
|
+
options.config = need(arg, argv[++i]);
|
|
474
|
+
break;
|
|
475
|
+
case "--api-url":
|
|
476
|
+
options.apiUrl = need(arg, argv[++i]);
|
|
477
|
+
break;
|
|
478
|
+
case "--api-key":
|
|
479
|
+
options.apiKey = need(arg, argv[++i]);
|
|
480
|
+
break;
|
|
481
|
+
case "--warn-days": {
|
|
482
|
+
const raw = need(arg, argv[++i]);
|
|
483
|
+
const n = Number(raw);
|
|
484
|
+
if (!Number.isFinite(n) || n < 0) throw new UsageError(`--warn-days must be a non-negative number, got "${raw}"`);
|
|
485
|
+
options.warnDays = Math.floor(n);
|
|
486
|
+
options.explicit.warnDays = true;
|
|
487
|
+
break;
|
|
488
|
+
}
|
|
489
|
+
case "--fail-on-warn":
|
|
490
|
+
options.failOnWarn = true;
|
|
491
|
+
options.explicit.failOnWarn = true;
|
|
492
|
+
break;
|
|
493
|
+
case "--fail-on-unknown":
|
|
494
|
+
options.failOnUnknown = true;
|
|
495
|
+
options.explicit.failOnUnknown = true;
|
|
496
|
+
break;
|
|
497
|
+
case "--no-policy":
|
|
498
|
+
options.usePolicy = false;
|
|
499
|
+
break;
|
|
500
|
+
case "--optimize":
|
|
501
|
+
options.optimize = true;
|
|
502
|
+
break;
|
|
503
|
+
case "--dry-run":
|
|
504
|
+
options.dryRun = true;
|
|
505
|
+
break;
|
|
506
|
+
case "--prune":
|
|
507
|
+
options.prune = true;
|
|
508
|
+
break;
|
|
509
|
+
case "--no-prune":
|
|
510
|
+
options.prune = false;
|
|
511
|
+
break;
|
|
512
|
+
case "--json":
|
|
513
|
+
options.json = true;
|
|
514
|
+
break;
|
|
515
|
+
case "--quiet":
|
|
516
|
+
case "-q":
|
|
517
|
+
options.quiet = true;
|
|
518
|
+
break;
|
|
519
|
+
case "--help":
|
|
520
|
+
case "-h":
|
|
521
|
+
options.command = "help";
|
|
522
|
+
return options;
|
|
523
|
+
default:
|
|
524
|
+
if (arg.startsWith("-")) throw new UsageError(`Unknown flag: ${arg}`);
|
|
525
|
+
positional.push(arg);
|
|
526
|
+
}
|
|
527
|
+
}
|
|
528
|
+
options.paths = positional;
|
|
529
|
+
return options;
|
|
530
|
+
}
|
|
531
|
+
var HELP_TEXT = `llmintel \u2014 gate and track AI model lifecycle state in CI/CD
|
|
532
|
+
|
|
533
|
+
Usage:
|
|
534
|
+
llmintel check [paths...] [options] Fail the build on retired/at-risk models
|
|
535
|
+
llmintel sync [paths...] [options] Auto-watch the models your code uses (paid)
|
|
536
|
+
|
|
537
|
+
check
|
|
538
|
+
Scans the given files/directories (and/or --models) for referenced AI model ids,
|
|
539
|
+
looks up their lifecycle state via the LLMIntel API, and exits non-zero when any
|
|
540
|
+
referenced model is retired (or past due). Deprecated/retiring models, or models
|
|
541
|
+
retiring within --warn-days, are reported as warnings.
|
|
542
|
+
|
|
543
|
+
The gate also applies your account's central policy (configured in the dashboard):
|
|
544
|
+
the baseline failOn set, warn window, and fail-on-unknown. Explicit local flags
|
|
545
|
+
override the policy; pass --no-policy to ignore it and use only local flags.
|
|
546
|
+
|
|
547
|
+
sync
|
|
548
|
+
Discovers the models referenced in your codebase (same scan as check) and registers
|
|
549
|
+
them as your account's watched set, so push alerts (webhook/Slack/PagerDuty) track
|
|
550
|
+
your real footprint with no manual upkeep. Run it in CI on merge to keep the set in
|
|
551
|
+
sync with your code. Replaces the watch set by default; use --no-prune to only add.
|
|
552
|
+
Requires a paid plan.
|
|
553
|
+
|
|
554
|
+
Options:
|
|
555
|
+
-m, --models <a,b,c> Explicit model ids/aliases (repeatable)
|
|
556
|
+
-c, --config <file> JSON config: { "models": [...], "paths": [...] }
|
|
557
|
+
--api-url <url> API base URL (env LLMINTEL_API_URL, default https://llmintel.vercel.app)
|
|
558
|
+
--api-key <key> API key (env LLMINTEL_API_KEY)
|
|
559
|
+
--warn-days <n> check: warn window in days (default 90)
|
|
560
|
+
--fail-on-warn check: exit non-zero on warnings too
|
|
561
|
+
--fail-on-unknown check: treat unresolved references as errors
|
|
562
|
+
--no-policy check: ignore the account's central policy (use local flags only)
|
|
563
|
+
--optimize check: also show advisory cheaper/faster model suggestions (paid; never fails the build)
|
|
564
|
+
--dry-run sync: print the diff without writing
|
|
565
|
+
--no-prune sync: only add models, never remove
|
|
566
|
+
--json Machine-readable JSON output
|
|
567
|
+
-q, --quiet Only print failures / changes
|
|
568
|
+
-h, --help Show this help
|
|
569
|
+
-v, --version Show version
|
|
570
|
+
|
|
571
|
+
Exit codes:
|
|
572
|
+
0 success (check: all healthy; sync: applied or dry-run)
|
|
573
|
+
1 check: one or more errors (retired / past due), or warnings with --fail-on-warn
|
|
574
|
+
2 usage/configuration error
|
|
575
|
+
3 API or network error
|
|
576
|
+
`;
|
|
577
|
+
|
|
578
|
+
// src/index.ts
|
|
579
|
+
import { pathToFileURL } from "url";
|
|
580
|
+
import { realpathSync } from "fs";
|
|
581
|
+
var EXIT_OK = 0;
|
|
582
|
+
var EXIT_USAGE = 2;
|
|
583
|
+
var EXIT_API = 3;
|
|
584
|
+
var VERSION = "0.1.1";
|
|
585
|
+
async function mergeConfig(options, env) {
|
|
586
|
+
let paths = options.paths;
|
|
587
|
+
let models = options.models;
|
|
588
|
+
let warnDays = options.warnDays;
|
|
589
|
+
let apiUrl = options.apiUrl;
|
|
590
|
+
if (options.config) {
|
|
591
|
+
const config = await loadConfig(options.config);
|
|
592
|
+
if (config.models) models = [...models, ...config.models];
|
|
593
|
+
if (config.paths) paths = [...paths, ...config.paths];
|
|
594
|
+
if (config.warnDays !== void 0 && options.warnDays === 90) warnDays = config.warnDays;
|
|
595
|
+
if (config.apiUrl && env.LLMINTEL_API_URL === void 0) apiUrl = config.apiUrl;
|
|
596
|
+
}
|
|
597
|
+
return { paths, models, warnDays, apiUrl };
|
|
598
|
+
}
|
|
599
|
+
async function gatherReferences(models, paths) {
|
|
600
|
+
const explicit = models.map((value) => ({ value, source: "--models" }));
|
|
601
|
+
const scanned = paths.length > 0 ? await scanPaths(paths) : [];
|
|
602
|
+
return dedupeReferences([explicit, scanned]);
|
|
603
|
+
}
|
|
604
|
+
async function run(deps) {
|
|
605
|
+
let options;
|
|
606
|
+
try {
|
|
607
|
+
options = parseArgs(deps.argv, deps.env);
|
|
608
|
+
} catch (cause) {
|
|
609
|
+
if (cause instanceof UsageError) {
|
|
610
|
+
deps.error(`error: ${cause.message}
|
|
611
|
+
`);
|
|
612
|
+
deps.error(HELP_TEXT);
|
|
613
|
+
return EXIT_USAGE;
|
|
614
|
+
}
|
|
615
|
+
throw cause;
|
|
616
|
+
}
|
|
617
|
+
if (options.command === "help") {
|
|
618
|
+
deps.log(HELP_TEXT);
|
|
619
|
+
return EXIT_OK;
|
|
620
|
+
}
|
|
621
|
+
if (options.command === "version") {
|
|
622
|
+
deps.log(VERSION);
|
|
623
|
+
return EXIT_OK;
|
|
624
|
+
}
|
|
625
|
+
let config;
|
|
626
|
+
try {
|
|
627
|
+
config = await mergeConfig(options, deps.env);
|
|
628
|
+
} catch (cause) {
|
|
629
|
+
if (cause instanceof ConfigError) {
|
|
630
|
+
deps.error(`error: ${cause.message}`);
|
|
631
|
+
return EXIT_USAGE;
|
|
632
|
+
}
|
|
633
|
+
throw cause;
|
|
634
|
+
}
|
|
635
|
+
if (!options.apiKey) {
|
|
636
|
+
deps.error("error: no API key. Pass --api-key or set LLMINTEL_API_KEY.");
|
|
637
|
+
return EXIT_USAGE;
|
|
638
|
+
}
|
|
639
|
+
if (config.models.length === 0 && config.paths.length === 0) {
|
|
640
|
+
deps.error("error: nothing to check. Pass file paths and/or --models.");
|
|
641
|
+
deps.error(HELP_TEXT);
|
|
642
|
+
return EXIT_USAGE;
|
|
643
|
+
}
|
|
644
|
+
const references = await gatherReferences(config.models, config.paths);
|
|
645
|
+
if (references.length === 0) {
|
|
646
|
+
deps.error("error: no model references found in the given paths.");
|
|
647
|
+
return EXIT_USAGE;
|
|
648
|
+
}
|
|
649
|
+
if (options.command === "sync") {
|
|
650
|
+
return runSync(deps, options, config, references);
|
|
651
|
+
}
|
|
652
|
+
return runCheck(deps, options, config, references);
|
|
653
|
+
}
|
|
654
|
+
async function runCheck(deps, options, config, references) {
|
|
655
|
+
const fetchModels = deps.fetchModels ?? fetchAllModels;
|
|
656
|
+
let apiModels;
|
|
657
|
+
try {
|
|
658
|
+
apiModels = await fetchModels({ baseUrl: config.apiUrl, apiKey: options.apiKey });
|
|
659
|
+
} catch (cause) {
|
|
660
|
+
if (cause instanceof ApiError) {
|
|
661
|
+
deps.error(`error: API request failed: ${cause.message}`);
|
|
662
|
+
return EXIT_API;
|
|
663
|
+
}
|
|
664
|
+
throw cause;
|
|
665
|
+
}
|
|
666
|
+
let warnDays = config.warnDays;
|
|
667
|
+
let failOnUnknown = options.failOnUnknown;
|
|
668
|
+
let failOn = ["retired"];
|
|
669
|
+
if (options.usePolicy) {
|
|
670
|
+
const getPolicy = deps.fetchPolicy ?? fetchPolicy;
|
|
671
|
+
try {
|
|
672
|
+
const policy = await getPolicy({ baseUrl: config.apiUrl, apiKey: options.apiKey });
|
|
673
|
+
failOn = policy.failOn.filter(
|
|
674
|
+
(s) => ["announced", "active", "legacy", "deprecated", "retiring", "retired"].includes(s)
|
|
675
|
+
);
|
|
676
|
+
if (failOn.length === 0) failOn = ["retired"];
|
|
677
|
+
if (!options.explicit.warnDays) warnDays = policy.warnWindowDays;
|
|
678
|
+
if (!options.explicit.failOnUnknown) failOnUnknown = policy.failOnUnknown;
|
|
679
|
+
} catch (cause) {
|
|
680
|
+
if (!options.quiet) {
|
|
681
|
+
const detail = cause instanceof ApiError ? cause.message : String(cause);
|
|
682
|
+
deps.error(`note: could not fetch account policy (${detail}); using local flags only.`);
|
|
683
|
+
}
|
|
684
|
+
}
|
|
685
|
+
}
|
|
686
|
+
const report = buildReport(references, apiModels, {
|
|
687
|
+
warnDays,
|
|
688
|
+
failOnWarn: options.failOnWarn,
|
|
689
|
+
failOnUnknown,
|
|
690
|
+
failOn
|
|
691
|
+
});
|
|
692
|
+
let nudges = [];
|
|
693
|
+
if (options.optimize) {
|
|
694
|
+
nudges = await gatherOptimizations(deps, options, config, references, apiModels);
|
|
695
|
+
}
|
|
696
|
+
if (options.json) {
|
|
697
|
+
deps.log(formatJson(report, nudges));
|
|
698
|
+
} else {
|
|
699
|
+
deps.log(formatHuman(report, options.quiet));
|
|
700
|
+
const opt = formatOptimization(nudges);
|
|
701
|
+
if (opt) deps.log(opt);
|
|
702
|
+
}
|
|
703
|
+
return report.exitCode;
|
|
704
|
+
}
|
|
705
|
+
var MAX_OPTIMIZE_LOOKUPS = 25;
|
|
706
|
+
async function gatherOptimizations(deps, options, config, references, apiModels) {
|
|
707
|
+
const getOptimization = deps.fetchOptimization ?? fetchOptimization;
|
|
708
|
+
const index = buildIndex(apiModels);
|
|
709
|
+
const activeIds = [];
|
|
710
|
+
const seen = /* @__PURE__ */ new Set();
|
|
711
|
+
for (const ref of references) {
|
|
712
|
+
const model = index.get(ref.value.toLowerCase());
|
|
713
|
+
if (!model || model.lifecycleState !== "active") continue;
|
|
714
|
+
if (seen.has(model.id)) continue;
|
|
715
|
+
seen.add(model.id);
|
|
716
|
+
activeIds.push(model.id);
|
|
717
|
+
if (activeIds.length >= MAX_OPTIMIZE_LOOKUPS) break;
|
|
718
|
+
}
|
|
719
|
+
const nudges = [];
|
|
720
|
+
for (const modelId of activeIds) {
|
|
721
|
+
try {
|
|
722
|
+
const optimization = await getOptimization({
|
|
723
|
+
baseUrl: config.apiUrl,
|
|
724
|
+
apiKey: options.apiKey,
|
|
725
|
+
modelId
|
|
726
|
+
});
|
|
727
|
+
if (!optimization) continue;
|
|
728
|
+
for (const c of optimization.candidates) {
|
|
729
|
+
nudges.push({
|
|
730
|
+
modelId,
|
|
731
|
+
candidateId: c.candidateId,
|
|
732
|
+
candidateDisplayName: c.candidateDisplayName,
|
|
733
|
+
candidateProvider: c.candidateProvider,
|
|
734
|
+
reasons: c.reasons,
|
|
735
|
+
crossProvider: c.crossProvider
|
|
736
|
+
});
|
|
737
|
+
}
|
|
738
|
+
} catch (cause) {
|
|
739
|
+
if (!options.quiet) {
|
|
740
|
+
const detail = cause instanceof ApiError ? cause.message : String(cause);
|
|
741
|
+
deps.error(`note: optimization lookup for ${modelId} failed (${detail}); skipping.`);
|
|
742
|
+
}
|
|
743
|
+
}
|
|
744
|
+
}
|
|
745
|
+
return nudges;
|
|
746
|
+
}
|
|
747
|
+
async function runSync(deps, options, config, references) {
|
|
748
|
+
const sync = deps.syncWatches ?? syncWatches;
|
|
749
|
+
const discovered = [...new Set(references.map((r) => r.value))];
|
|
750
|
+
if (options.dryRun) {
|
|
751
|
+
const summary2 = {
|
|
752
|
+
added: discovered.sort(),
|
|
753
|
+
removed: [],
|
|
754
|
+
unchanged: [],
|
|
755
|
+
unresolved: [],
|
|
756
|
+
dryRun: true,
|
|
757
|
+
discovered: discovered.length
|
|
758
|
+
};
|
|
759
|
+
deps.log(options.json ? formatSyncJson(summary2) : formatSyncHuman(summary2, options.quiet));
|
|
760
|
+
return EXIT_OK;
|
|
761
|
+
}
|
|
762
|
+
let result;
|
|
763
|
+
try {
|
|
764
|
+
result = await sync({
|
|
765
|
+
baseUrl: config.apiUrl,
|
|
766
|
+
apiKey: options.apiKey,
|
|
767
|
+
models: discovered,
|
|
768
|
+
prune: options.prune
|
|
769
|
+
});
|
|
770
|
+
} catch (cause) {
|
|
771
|
+
if (cause instanceof ApiError) {
|
|
772
|
+
deps.error(`error: API request failed: ${cause.message}`);
|
|
773
|
+
return EXIT_API;
|
|
774
|
+
}
|
|
775
|
+
throw cause;
|
|
776
|
+
}
|
|
777
|
+
const summary = { ...result, dryRun: false, discovered: discovered.length };
|
|
778
|
+
deps.log(options.json ? formatSyncJson(summary) : formatSyncHuman(summary, options.quiet));
|
|
779
|
+
return EXIT_OK;
|
|
780
|
+
}
|
|
781
|
+
async function main() {
|
|
782
|
+
const code = await run({
|
|
783
|
+
argv: process.argv.slice(2),
|
|
784
|
+
env: process.env,
|
|
785
|
+
log: (msg) => process.stdout.write(`${msg}
|
|
786
|
+
`),
|
|
787
|
+
error: (msg) => process.stderr.write(`${msg}
|
|
788
|
+
`)
|
|
789
|
+
});
|
|
790
|
+
process.exit(code);
|
|
791
|
+
}
|
|
792
|
+
var isEntrypoint = (() => {
|
|
793
|
+
const entry = process.argv[1];
|
|
794
|
+
if (!entry) return false;
|
|
795
|
+
try {
|
|
796
|
+
return import.meta.url === pathToFileURL(realpathSync(entry)).href;
|
|
797
|
+
} catch {
|
|
798
|
+
return false;
|
|
799
|
+
}
|
|
800
|
+
})();
|
|
801
|
+
if (isEntrypoint) {
|
|
802
|
+
void main();
|
|
803
|
+
}
|
|
804
|
+
export {
|
|
805
|
+
run
|
|
806
|
+
};
|
package/package.json
ADDED
|
@@ -0,0 +1,53 @@
|
|
|
1
|
+
{
|
|
2
|
+
"name": "@llmintel/cli",
|
|
3
|
+
"version": "0.2.0",
|
|
4
|
+
"description": "Fail your CI build when it references retired or soon-to-be-retired AI models. The LLMIntel lifecycle gate.",
|
|
5
|
+
"license": "MIT",
|
|
6
|
+
"type": "module",
|
|
7
|
+
"bin": {
|
|
8
|
+
"llmintel": "dist/index.js"
|
|
9
|
+
},
|
|
10
|
+
"files": [
|
|
11
|
+
"dist",
|
|
12
|
+
"action.yml",
|
|
13
|
+
"README.md"
|
|
14
|
+
],
|
|
15
|
+
"keywords": [
|
|
16
|
+
"ai",
|
|
17
|
+
"llm",
|
|
18
|
+
"deprecation",
|
|
19
|
+
"lifecycle",
|
|
20
|
+
"ci",
|
|
21
|
+
"cd",
|
|
22
|
+
"openai",
|
|
23
|
+
"anthropic",
|
|
24
|
+
"model"
|
|
25
|
+
],
|
|
26
|
+
"repository": {
|
|
27
|
+
"type": "git",
|
|
28
|
+
"url": "git+https://github.com/hivemindunit/llmintel-cli.git",
|
|
29
|
+
"directory": "packages/cli"
|
|
30
|
+
},
|
|
31
|
+
"homepage": "https://llmintel.vercel.app/docs",
|
|
32
|
+
"bugs": "https://github.com/hivemindunit/llmintel-cli/issues",
|
|
33
|
+
"publishConfig": {
|
|
34
|
+
"access": "public"
|
|
35
|
+
},
|
|
36
|
+
"engines": {
|
|
37
|
+
"node": ">=20"
|
|
38
|
+
},
|
|
39
|
+
"scripts": {
|
|
40
|
+
"build": "tsup",
|
|
41
|
+
"typecheck": "tsc --noEmit",
|
|
42
|
+
"check": "tsx src/index.ts check",
|
|
43
|
+
"prepublishOnly": "tsup"
|
|
44
|
+
},
|
|
45
|
+
"dependencies": {},
|
|
46
|
+
"devDependencies": {
|
|
47
|
+
"@llmintel/schema": "workspace:*",
|
|
48
|
+
"tsup": "^8.3.5",
|
|
49
|
+
"tsx": "^4.19.2",
|
|
50
|
+
"typescript": "^5.7.2",
|
|
51
|
+
"vitest": "^2.1.8"
|
|
52
|
+
}
|
|
53
|
+
}
|