completion-kit 0.27.5 → 0.27.7
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 69f57e2ec1d9363a088d85a4ecf6c3632c1f971e08a8df351c0b2bbbd9b1ba16
|
|
4
|
+
data.tar.gz: 8becf1e826e0a1dd537af4d4e762154715d693a719bd97a7464c2d54beb4fe78
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: ebb8b90edfe00dbffc9b7339594fefb8884adbf04fa1e26bc6a42dcae66e17aaf3d77c03b873d8d5e91ab4edb6dbcb3e4c3b142c17170a4860f3b307510907cd
|
|
7
|
+
data.tar.gz: 5ec296ba854f8ad3162f67303d8898d1db83831ad9a443b5b20acf473b53c7ee97bb74d8ca4a213e0a27eb5fe6b55e68bb39c2efc7379e44029224337f45e269
|
data/README.md
CHANGED
|
@@ -86,6 +86,7 @@ CompletionKit discovers available models from each provider's API automatically.
|
|
|
86
86
|
| **Anthropic** | `ANTHROPIC_API_KEY` | Claude Opus, Sonnet, Haiku |
|
|
87
87
|
| **Ollama / local endpoint** | `OLLAMA_API_ENDPOINT` (default: `http://localhost:11434/v1`) | Any model you've `ollama pull`-ed, or any OpenAI-compatible local server (vLLM, LM Studio, llama.cpp) |
|
|
88
88
|
| **OpenRouter** | `OPENROUTER_API_KEY` | 100+ models from 30+ providers through one API key |
|
|
89
|
+
| **Azure AI Foundry** | No env var — add via the Providers UI / a `ProviderCredential` (`api_key` + `api_endpoint` required, `api_version` optional) | Azure OpenAI deployments and Foundry projects |
|
|
89
90
|
|
|
90
91
|
Set these as environment variables or configure them in the generated initializer. You can also add provider credentials through the web UI under Settings.
|
|
91
92
|
|
|
@@ -5,7 +5,7 @@ module CompletionKit
|
|
|
5
5
|
|
|
6
6
|
TOOLS = {
|
|
7
7
|
"judges_replay" => {
|
|
8
|
-
description: "
|
|
8
|
+
description: "Create a scoring run for the current judge over a dataset's existing outputs (wraps runs_create with prompt_id omitted and output_column supplied). This only sets up the run; call runs_generate to actually re-judge the outputs so you can compare against human verdicts.",
|
|
9
9
|
inputSchema: {
|
|
10
10
|
type: "object",
|
|
11
11
|
properties: {
|
|
@@ -20,7 +20,7 @@ module CompletionKit
|
|
|
20
20
|
handler: :replay
|
|
21
21
|
},
|
|
22
22
|
"judges_compare" => {
|
|
23
|
-
description: "Compare two metric
|
|
23
|
+
description: "Compare two versions of one metric's agreement stats side by side. Requires metric_id, metric_version_a_id, and metric_version_b_id (both versions must belong to that metric). Unavailable for check metrics.",
|
|
24
24
|
inputSchema: {
|
|
25
25
|
type: "object",
|
|
26
26
|
properties: {
|
|
@@ -30,7 +30,7 @@
|
|
|
30
30
|
<label for="ck-tab-datasets" class="ck-api-tabs__label">Datasets <span class="ck-api-tabs__count">5</span></label>
|
|
31
31
|
<label for="ck-tab-metrics" class="ck-api-tabs__label">Metrics <span class="ck-api-tabs__count">10</span></label>
|
|
32
32
|
<label for="ck-tab-metric-groups" class="ck-api-tabs__label">Metric Groups <span class="ck-api-tabs__count">5</span></label>
|
|
33
|
-
<label for="ck-tab-agreements" class="ck-api-tabs__label">Agreements <span class="ck-api-tabs__count">
|
|
33
|
+
<label for="ck-tab-agreements" class="ck-api-tabs__label">Agreements <span class="ck-api-tabs__count">4</span></label>
|
|
34
34
|
<label for="ck-tab-tags" class="ck-api-tabs__label">Tags <span class="ck-api-tabs__count">5</span></label>
|
|
35
35
|
<label for="ck-tab-providers" class="ck-api-tabs__label">Providers <span class="ck-api-tabs__count">5</span></label>
|
|
36
36
|
<label for="ck-tab-imports" class="ck-api-tabs__label">Imports <span class="ck-api-tabs__count">1</span></label>
|
|
@@ -83,7 +83,7 @@
|
|
|
83
83
|
<p class="ck-copy">Create, version, and manage LLM prompt templates.</p>
|
|
84
84
|
<div class="ck-api-endpoint">
|
|
85
85
|
<p class="ck-api-method"><span class="ck-chip ck-chip--soft">GET</span> /api/v1/prompts</p>
|
|
86
|
-
<p class="ck-meta-copy">List all prompts, ordered by most recent.</p>
|
|
86
|
+
<p class="ck-meta-copy">List all prompts, ordered by most recent. Supports pagination (<code>limit</code>, <code>offset</code>) and <code>tag[]</code> filtering.</p>
|
|
87
87
|
<%= render "completion_kit/api_reference/example", base_url: base_url, token: token, real_token: real_token, cmd: "curl #{base_url}/api/v1/prompts \\\n -H \"Authorization: Bearer #{token}\"" %>
|
|
88
88
|
</div>
|
|
89
89
|
<div class="ck-api-endpoint">
|
|
@@ -94,7 +94,7 @@
|
|
|
94
94
|
</div>
|
|
95
95
|
<div class="ck-api-endpoint">
|
|
96
96
|
<p class="ck-api-method"><span class="ck-chip ck-chip--soft">GET</span> /api/v1/prompts/:id</p>
|
|
97
|
-
<p class="ck-meta-copy">Get a single prompt by ID.</p>
|
|
97
|
+
<p class="ck-meta-copy">Get a single prompt by numeric ID or slug.</p>
|
|
98
98
|
</div>
|
|
99
99
|
<%= render "completion_kit/api_reference/resource_list", title: "Your published prompts",
|
|
100
100
|
items: published_prompts.map { |p|
|
|
@@ -103,7 +103,8 @@
|
|
|
103
103
|
} %>
|
|
104
104
|
<div class="ck-api-endpoint">
|
|
105
105
|
<p class="ck-api-method"><span class="ck-chip ck-chip--soft">PATCH</span> /api/v1/prompts/:id</p>
|
|
106
|
-
<p class="ck-meta-copy">Update a prompt. Accepts same params as create.</p>
|
|
106
|
+
<p class="ck-meta-copy">Update a prompt. Accepts same params as create. If the prompt has no runs it is updated in place; if it already has runs, a new published version is created and returned (with a new id), leaving the original unchanged.</p>
|
|
107
|
+
<%= render "completion_kit/api_reference/example", base_url: base_url, token: token, real_token: real_token, cmd: "curl -X PATCH #{base_url}/api/v1/prompts/1 \\\n -H \"Authorization: Bearer #{token}\" \\\n -H \"Content-Type: application/json\" \\\n -d '{\"template\": \"Summarize concisely: {{text}}\"}'" %>
|
|
107
108
|
</div>
|
|
108
109
|
<div class="ck-api-endpoint">
|
|
109
110
|
<p class="ck-api-method"><span class="ck-chip" style="color: var(--ck-danger);">DELETE</span> /api/v1/prompts/:id</p>
|
|
@@ -112,6 +113,7 @@
|
|
|
112
113
|
<div class="ck-api-endpoint">
|
|
113
114
|
<p class="ck-api-method"><span class="ck-chip ck-chip--soft">POST</span> /api/v1/prompts/:id/publish</p>
|
|
114
115
|
<p class="ck-meta-copy">Publish a prompt version, making it the current version in its family.</p>
|
|
116
|
+
<%= render "completion_kit/api_reference/example", base_url: base_url, token: token, real_token: real_token, cmd: "curl -X POST #{base_url}/api/v1/prompts/1/publish \\\n -H \"Authorization: Bearer #{token}\"" %>
|
|
115
117
|
</div>
|
|
116
118
|
</div>
|
|
117
119
|
|
|
@@ -122,11 +124,12 @@
|
|
|
122
124
|
<p class="ck-api-method"><span class="ck-chip ck-chip--soft">GET</span> /api/v1/runs</p>
|
|
123
125
|
<p class="ck-meta-copy">List runs with response counts and average scores. Supports pagination (<code>limit</code>, <code>offset</code>) and the following filters.</p>
|
|
124
126
|
<p class="ck-api-params"><strong>Optional filters:</strong> <code>status</code> (<code>pending</code>, <code>running</code>, <code>completed</code>, <code>failed</code>), <code>prompt_id</code>, <code>dataset_id</code>, <code>tag[]</code></p>
|
|
127
|
+
<%= render "completion_kit/api_reference/example", base_url: base_url, token: token, real_token: real_token, cmd: "curl \"#{base_url}/api/v1/runs?status=completed&prompt_id=1&tag[]=real%20estate\" \\\n -H \"Authorization: Bearer #{token}\"" %>
|
|
125
128
|
</div>
|
|
126
129
|
<div class="ck-api-endpoint">
|
|
127
130
|
<p class="ck-api-method"><span class="ck-chip ck-chip--soft">POST</span> /api/v1/runs</p>
|
|
128
131
|
<p class="ck-meta-copy">Create a new run.</p>
|
|
129
|
-
<p class="ck-api-params"><strong>Optional:</strong> <code>name</code>, <code>prompt_id</code>, <code>dataset_id</code>, <code>metric_ids</code>, <code>judge_model</code>, <code>output_column</code> (score existing outputs: omit <code>prompt_id</code> and grade a dataset column instead, default <code>actual_output</code>), <code>expected_column</code> (dataset column holding each row's answer key, given to the judge and to checks that compare against the row's expected value, default <code>expected_output</code>)</p>
|
|
132
|
+
<p class="ck-api-params"><strong>Optional:</strong> <code>name</code>, <code>prompt_id</code>, <code>dataset_id</code>, <code>metric_ids</code>, <code>judge_model</code>, <code>temperature</code>, <code>output_column</code> (score existing outputs: omit <code>prompt_id</code> and grade a dataset column instead, default <code>actual_output</code>), <code>expected_column</code> (dataset column holding each row's answer key, given to the judge and to checks that compare against the row's expected value, default <code>expected_output</code>)</p>
|
|
130
133
|
<%= render "completion_kit/api_reference/example", base_url: base_url, token: token, real_token: real_token, cmd: "curl -X POST #{base_url}/api/v1/runs \\\n -H \"Authorization: Bearer #{token}\" \\\n -H \"Content-Type: application/json\" \\\n -d '{\"prompt_id\": 1, \"dataset_id\": 1, \"metric_ids\": [1, 2]}'" %>
|
|
131
134
|
</div>
|
|
132
135
|
<div class="ck-api-endpoint">
|
|
@@ -146,25 +149,29 @@
|
|
|
146
149
|
</div>
|
|
147
150
|
<div class="ck-api-endpoint">
|
|
148
151
|
<p class="ck-api-method"><span class="ck-chip ck-chip--soft">POST</span> /api/v1/runs/:id/retry_failures</p>
|
|
149
|
-
<p class="ck-meta-copy">Re-queue
|
|
152
|
+
<p class="ck-meta-copy">Re-queue responses that failed during generation. Returns 202 Accepted, or 409 Conflict if the judge version has drifted since the run executed (use <code>/rerun</code> instead to avoid mixing versions in one run).</p>
|
|
153
|
+
<p class="ck-api-params"><strong>Optional:</strong> <code>only</code> (array of failed response IDs, to retry just a subset)</p>
|
|
154
|
+
<%= render "completion_kit/api_reference/example", base_url: base_url, token: token, real_token: real_token, cmd: "curl -X POST #{base_url}/api/v1/runs/1/retry_failures \\\n -H \"Authorization: Bearer #{token}\"" %>
|
|
150
155
|
</div>
|
|
151
156
|
<div class="ck-api-endpoint">
|
|
152
157
|
<p class="ck-api-method"><span class="ck-chip ck-chip--soft">POST</span> /api/v1/runs/:id/rerun</p>
|
|
153
|
-
<p class="ck-meta-copy">Clone the run and start generating responses on the copy against the current prompt and metric versions. Returns the new run with
|
|
158
|
+
<p class="ck-meta-copy">Clone the run and start generating responses on the copy against the current prompt and metric versions. Returns the new run with 202 Accepted. Useful for capturing a fresh baseline after metric edits.</p>
|
|
154
159
|
<%= render "completion_kit/api_reference/example", base_url: base_url, token: token, real_token: real_token, cmd: "curl -X POST #{base_url}/api/v1/runs/1/rerun \\\n -H \"Authorization: Bearer #{token}\"" %>
|
|
155
160
|
</div>
|
|
156
161
|
<div class="ck-api-endpoint">
|
|
157
162
|
<p class="ck-api-method"><span class="ck-chip ck-chip--soft">POST</span> /api/v1/runs/:id/regrade</p>
|
|
158
163
|
<p class="ck-meta-copy">Re-judge the existing successful responses against the current metric versions without regenerating model output. Returns 202 Accepted, or 422 if no responses are eligible.</p>
|
|
164
|
+
<%= render "completion_kit/api_reference/example", base_url: base_url, token: token, real_token: real_token, cmd: "curl -X POST #{base_url}/api/v1/runs/1/regrade \\\n -H \"Authorization: Bearer #{token}\"" %>
|
|
159
165
|
</div>
|
|
160
166
|
<div class="ck-api-endpoint">
|
|
161
167
|
<p class="ck-api-method"><span class="ck-chip ck-chip--soft">GET</span> /api/v1/runs/:id/compare?with=:other_id</p>
|
|
162
|
-
<p class="ck-meta-copy">Side-by-side comparison against another run. Returns <code>{
|
|
168
|
+
<p class="ck-meta-copy">Side-by-side comparison against another run. Returns <code>{left_run_id, right_run_id, metric_ids: [...], rows: [...]}</code> with one row per input case, per-metric scores on both sides, and the delta. Rows are keyed by the left run's input cases: a case present only in the left run is returned with the right side nulled out, and a case present only in the right run is omitted.</p>
|
|
163
169
|
<%= render "completion_kit/api_reference/example", base_url: base_url, token: token, real_token: real_token, cmd: "curl \"#{base_url}/api/v1/runs/1/compare?with=2\" \\\n -H \"Authorization: Bearer #{token}\"" %>
|
|
164
170
|
</div>
|
|
165
171
|
<div class="ck-api-endpoint">
|
|
166
172
|
<p class="ck-api-method"><span class="ck-chip ck-chip--soft">PATCH</span> /api/v1/runs/:id</p>
|
|
167
173
|
<p class="ck-meta-copy">Update a run. Accepts same params as create.</p>
|
|
174
|
+
<%= render "completion_kit/api_reference/example", base_url: base_url, token: token, real_token: real_token, cmd: "curl -X PATCH #{base_url}/api/v1/runs/1 \\\n -H \"Authorization: Bearer #{token}\" \\\n -H \"Content-Type: application/json\" \\\n -d '{\"name\": \"baseline v2\", \"metric_ids\": [1, 2, 3]}'" %>
|
|
168
175
|
</div>
|
|
169
176
|
<div class="ck-api-endpoint">
|
|
170
177
|
<p class="ck-api-method"><span class="ck-chip" style="color: var(--ck-danger);">DELETE</span> /api/v1/runs/:id</p>
|
|
@@ -178,7 +185,7 @@
|
|
|
178
185
|
<div class="ck-api-endpoint">
|
|
179
186
|
<p class="ck-api-method"><span class="ck-chip ck-chip--soft">GET</span> /api/v1/runs/:run_id/responses</p>
|
|
180
187
|
<p class="ck-meta-copy">List responses for a run, including nested review scores.</p>
|
|
181
|
-
<p class="ck-api-params"><strong>Optional filters:</strong> <code>status</code> (<code>pending</code>, <code>succeeded</code>, <code>failed</code>), plus <code>limit</code> and <code>offset</code></p>
|
|
188
|
+
<p class="ck-api-params"><strong>Optional filters:</strong> <code>status</code> (<code>pending</code>, <code>retrying</code>, <code>succeeded</code>, <code>failed</code>), plus <code>limit</code> and <code>offset</code></p>
|
|
182
189
|
<%= render "completion_kit/api_reference/example", base_url: base_url, token: token, real_token: real_token, cmd: "curl #{base_url}/api/v1/runs/1/responses \\\n -H \"Authorization: Bearer #{token}\"" %>
|
|
183
190
|
</div>
|
|
184
191
|
<div class="ck-api-endpoint">
|
|
@@ -210,6 +217,7 @@
|
|
|
210
217
|
<div class="ck-api-endpoint">
|
|
211
218
|
<p class="ck-api-method"><span class="ck-chip ck-chip--soft">GET</span> <span class="ck-chip ck-chip--soft">PATCH</span> <span class="ck-chip" style="color: var(--ck-danger);">DELETE</span> /api/v1/datasets/:id</p>
|
|
212
219
|
<p class="ck-meta-copy">Get, update, or delete a dataset.</p>
|
|
220
|
+
<%= render "completion_kit/api_reference/example", base_url: base_url, token: token, real_token: real_token, cmd: "curl -X PATCH #{base_url}/api/v1/datasets/1 \\\n -H \"Authorization: Bearer #{token}\" \\\n -H \"Content-Type: application/json\" \\\n -d '{\"name\": \"tickets-q3\"}'" %>
|
|
213
221
|
</div>
|
|
214
222
|
<%= render "completion_kit/api_reference/resource_list", title: "Your datasets",
|
|
215
223
|
items: datasets.map { |d|
|
|
@@ -234,6 +242,7 @@
|
|
|
234
242
|
<div class="ck-api-endpoint">
|
|
235
243
|
<p class="ck-api-method"><span class="ck-chip ck-chip--soft">GET</span> <span class="ck-chip ck-chip--soft">PATCH</span> <span class="ck-chip" style="color: var(--ck-danger);">DELETE</span> /api/v1/metrics/:id</p>
|
|
236
244
|
<p class="ck-meta-copy">Get, update, or delete a metric.</p>
|
|
245
|
+
<%= render "completion_kit/api_reference/example", base_url: base_url, token: token, real_token: real_token, cmd: "curl -X PATCH #{base_url}/api/v1/metrics/1 \\\n -H \"Authorization: Bearer #{token}\" \\\n -H \"Content-Type: application/json\" \\\n -d '{\"instruction\": \"Is the response factually accurate?\", \"rubric_bands\": [{\"stars\": 5, \"description\": \"fully accurate\"}]}'" %>
|
|
237
246
|
</div>
|
|
238
247
|
<%= render "completion_kit/api_reference/resource_list", title: "Your metrics",
|
|
239
248
|
items: metrics.map { |m|
|
|
@@ -247,7 +256,7 @@
|
|
|
247
256
|
</div>
|
|
248
257
|
<div class="ck-api-endpoint">
|
|
249
258
|
<p class="ck-api-method"><span class="ck-chip ck-chip--soft">POST</span> /api/v1/metrics/:id/suggest_variants</p>
|
|
250
|
-
<p class="ck-meta-copy">Generate draft metric versions from the current disagreements. Returns 201 with the new draft versions, 422 if no disagreements exist or the model produced nothing usable.</p>
|
|
259
|
+
<p class="ck-meta-copy">Generate draft metric versions from the current disagreements. Only rubric-style (llm_judge) metrics can suggest variants. Returns 201 with the new draft versions, or 422 if the metric is a check (checks are exact and have no variants to suggest), if no disagreements exist, or if the model produced nothing usable.</p>
|
|
251
260
|
<p class="ck-api-params"><strong>Optional:</strong> <code>count</code>, <code>model</code></p>
|
|
252
261
|
<%= render "completion_kit/api_reference/example", base_url: base_url, token: token, real_token: real_token, cmd: "curl -X POST #{base_url}/api/v1/metrics/1/suggest_variants \\\n -H \"Authorization: Bearer #{token}\"" %>
|
|
253
262
|
</div>
|
|
@@ -267,6 +276,7 @@
|
|
|
267
276
|
<div class="ck-api-endpoint">
|
|
268
277
|
<p class="ck-api-method"><span class="ck-chip ck-chip--soft">POST</span> /api/v1/metrics/:metric_id/metric_versions/:id/publish</p>
|
|
269
278
|
<p class="ck-meta-copy">Publish the version as current. Works for a draft (promote) or a superseded published version (revert). Copies the version's instruction and rubric (or check_config for checks) back onto the metric.</p>
|
|
279
|
+
<%= render "completion_kit/api_reference/example", base_url: base_url, token: token, real_token: real_token, cmd: "curl -X POST #{base_url}/api/v1/metrics/1/metric_versions/5/publish \\\n -H \"Authorization: Bearer #{token}\"" %>
|
|
270
280
|
</div>
|
|
271
281
|
<div class="ck-api-endpoint">
|
|
272
282
|
<p class="ck-api-method"><span class="ck-chip" style="color: var(--ck-danger);">DELETE</span> /api/v1/metrics/:metric_id/metric_versions/:id</p>
|
|
@@ -284,11 +294,13 @@
|
|
|
284
294
|
<div class="ck-api-endpoint">
|
|
285
295
|
<p class="ck-api-method"><span class="ck-chip ck-chip--soft">POST</span> /api/v1/metric_groups</p>
|
|
286
296
|
<p class="ck-meta-copy">Create a metric group.</p>
|
|
287
|
-
<p class="ck-api-params"><strong>Required:</strong> <code>name</code> <strong>Optional:</strong> <code>description</code>, <code>metric_ids</code> (array)</p>
|
|
297
|
+
<p class="ck-api-params"><strong>Required:</strong> <code>name</code> <strong>Optional:</strong> <code>description</code>, <code>metric_ids</code> (array), <code>tag_names</code> (array)</p>
|
|
298
|
+
<%= render "completion_kit/api_reference/example", base_url: base_url, token: token, real_token: real_token, cmd: "curl -X POST #{base_url}/api/v1/metric_groups \\\n -H \"Authorization: Bearer #{token}\" \\\n -H \"Content-Type: application/json\" \\\n -d '{\"name\": \"core quality\", \"metric_ids\": [1, 2, 3]}'" %>
|
|
288
299
|
</div>
|
|
289
300
|
<div class="ck-api-endpoint">
|
|
290
301
|
<p class="ck-api-method"><span class="ck-chip ck-chip--soft">GET</span> <span class="ck-chip ck-chip--soft">PATCH</span> <span class="ck-chip" style="color: var(--ck-danger);">DELETE</span> /api/v1/metric_groups/:id</p>
|
|
291
302
|
<p class="ck-meta-copy">Get, update, or delete a metric group. PATCH with <code>metric_ids</code> replaces all metric associations.</p>
|
|
303
|
+
<%= render "completion_kit/api_reference/example", base_url: base_url, token: token, real_token: real_token, cmd: "curl -X PATCH #{base_url}/api/v1/metric_groups/1 \\\n -H \"Authorization: Bearer #{token}\" \\\n -H \"Content-Type: application/json\" \\\n -d '{\"metric_ids\": [4, 5]}'" %>
|
|
292
304
|
</div>
|
|
293
305
|
<%= render "completion_kit/api_reference/resource_list", title: "Your metric groups",
|
|
294
306
|
items: metric_groups.map { |g|
|
|
@@ -300,6 +312,7 @@
|
|
|
300
312
|
<div class="ck-api-tabs__panel">
|
|
301
313
|
<h2 class="ck-section-title">Agreements</h2>
|
|
302
314
|
<p class="ck-copy">Per-verdict feedback events on a response/metric pair: agree, disagree (with a corrected score and note), or borderline. Agreements capture the metric version that was current when the verdict was cast, which is what drives the trust signal and the "stale" indicators across the rest of the API.</p>
|
|
315
|
+
<p class="ck-meta-copy">All agreement endpoints require the <code>judge_agreement_enabled</code> configuration (on by default). When it is disabled every agreement action returns <code>404</code> with <code>{"error": "Agreement disabled"}</code>.</p>
|
|
303
316
|
<div class="ck-api-endpoint">
|
|
304
317
|
<p class="ck-api-method"><span class="ck-chip ck-chip--soft">GET</span> /api/v1/agreements</p>
|
|
305
318
|
<p class="ck-meta-copy">List agreements across all runs. Supports filtering by any combination of the query params below.</p>
|
|
@@ -308,10 +321,14 @@
|
|
|
308
321
|
</div>
|
|
309
322
|
<div class="ck-api-endpoint">
|
|
310
323
|
<p class="ck-api-method"><span class="ck-chip ck-chip--soft">POST</span> /api/v1/runs/:run_id/responses/:response_id/metrics/:metric_id/agreements</p>
|
|
311
|
-
<p class="ck-meta-copy">Cast an agreement on a specific response/metric pair. The metric version on the record is set automatically from the run's review.</p>
|
|
312
|
-
<p class="ck-api-params"><strong>Required:</strong> <code>verdict</code>, <code>
|
|
324
|
+
<p class="ck-meta-copy">Cast an agreement on a specific response/metric pair. The metric version on the record is set automatically from the run's review. Idempotent per <code>(response, metric, created_by)</code>: the first cast creates the agreement (201 Created), and re-casting for the same <code>created_by</code> updates it in place (200 OK) rather than creating a duplicate.</p>
|
|
325
|
+
<p class="ck-api-params"><strong>Required:</strong> <code>verdict</code> (<code>agree</code>, <code>disagree</code>, or <code>borderline</code>); <code>corrected_score</code> (1–5) is required when <code>verdict</code> is <code>disagree</code> <strong>Optional:</strong> <code>corrected_score</code> (for <code>agree</code>/<code>borderline</code>), <code>note</code>, <code>created_by</code> (defaults to <code>"api"</code>)</p>
|
|
313
326
|
<%= render "completion_kit/api_reference/example", base_url: base_url, token: token, real_token: real_token, cmd: "curl -X POST #{base_url}/api/v1/runs/1/responses/42/metrics/3/agreements \\\n -H \"Authorization: Bearer #{token}\" \\\n -H \"Content-Type: application/json\" \\\n -d '{\"verdict\": \"disagree\", \"corrected_score\": 3, \"note\": \"too generous\", \"created_by\": \"alice\"}'" %>
|
|
314
327
|
</div>
|
|
328
|
+
<div class="ck-api-endpoint">
|
|
329
|
+
<p class="ck-api-method"><span class="ck-chip ck-chip--soft">GET</span> /api/v1/runs/:run_id/responses/:response_id/metrics/:metric_id/agreements</p>
|
|
330
|
+
<p class="ck-meta-copy">List agreements scoped to a specific run, response, and metric — the nested equivalent of the top-level index filtered by those three ids.</p>
|
|
331
|
+
</div>
|
|
315
332
|
<div class="ck-api-endpoint">
|
|
316
333
|
<p class="ck-api-method"><span class="ck-chip" style="color: var(--ck-danger);">DELETE</span> /api/v1/agreements/:id</p>
|
|
317
334
|
<p class="ck-meta-copy">Delete an agreement. Returns 204 No Content.</p>
|
|
@@ -320,7 +337,7 @@
|
|
|
320
337
|
|
|
321
338
|
<div class="ck-api-tabs__panel">
|
|
322
339
|
<h2 class="ck-section-title">Tags</h2>
|
|
323
|
-
<p class="ck-copy">Domain labels you can attach to metrics, prompts, runs, and
|
|
340
|
+
<p class="ck-copy">Domain labels you can attach to metrics, prompts, runs, datasets, and metric groups. Tags are auto-assigned a color from a 10-color palette. Each index page can be filtered by one or more tags using <code>?tag[]=name</code> query params (OR semantics).</p>
|
|
324
341
|
<div class="ck-api-endpoint">
|
|
325
342
|
<p class="ck-api-method"><span class="ck-chip ck-chip--soft">GET</span> /api/v1/tags</p>
|
|
326
343
|
<p class="ck-meta-copy">List all tags with name and color.</p>
|
|
@@ -343,7 +360,7 @@
|
|
|
343
360
|
} %>
|
|
344
361
|
<div class="ck-api-endpoint" style="padding-top: 1rem;">
|
|
345
362
|
<p class="ck-kicker" style="margin-bottom: 0.5rem;">Tagging resources</p>
|
|
346
|
-
<p class="ck-meta-copy">Metrics, prompts, runs, and
|
|
363
|
+
<p class="ck-meta-copy">Metrics, prompts, runs, datasets, and metric groups accept a <code>tag_names</code> array on their create and update endpoints. Passing a name that does not yet exist silently creates the tag. On PATCH, the list replaces all existing tags for that record (omit the field to leave tags unchanged).</p>
|
|
347
364
|
<%= render "completion_kit/api_reference/example", base_url: base_url, token: token, real_token: real_token, cmd: "curl -X POST #{base_url}/api/v1/metrics \\\n -H \"Authorization: Bearer #{token}\" \\\n -H \"Content-Type: application/json\" \\\n -d '{\"name\": \"Accuracy\", \"tag_names\": [\"real estate\"]}'" %>
|
|
348
365
|
</div>
|
|
349
366
|
<div class="ck-api-endpoint" style="padding-top: 1rem;">
|
|
@@ -355,7 +372,7 @@
|
|
|
355
372
|
<div class="ck-mcp-tool"><code class="ck-mcp-tool__name">tags_update</code><span class="ck-mcp-tool__desc">Update a tag's name</span></div>
|
|
356
373
|
<div class="ck-mcp-tool"><code class="ck-mcp-tool__name">tags_delete</code><span class="ck-mcp-tool__desc">Delete a tag and remove all its taggings</span></div>
|
|
357
374
|
</div>
|
|
358
|
-
<p class="ck-meta-copy" style="margin-top: 0.75rem;">The existing <code>metrics_create</code>, <code>metrics_update</code>, <code>prompts_create</code>, <code>prompts_update</code>, <code>runs_create</code>, <code>runs_update</code>, <code>datasets_create</code>,
|
|
375
|
+
<p class="ck-meta-copy" style="margin-top: 0.75rem;">The existing <code>metrics_create</code>, <code>metrics_update</code>, <code>prompts_create</code>, <code>prompts_update</code>, <code>runs_create</code>, <code>runs_update</code>, <code>datasets_create</code>, <code>datasets_create_from_url</code>, <code>datasets_update</code>, <code>metric_groups_create</code>, and <code>metric_groups_update</code> tools all accept a <code>tag_names</code> parameter with the same auto-create and replace semantics as the REST API.</p>
|
|
359
376
|
</div>
|
|
360
377
|
</div>
|
|
361
378
|
|
|
@@ -368,12 +385,15 @@
|
|
|
368
385
|
</div>
|
|
369
386
|
<div class="ck-api-endpoint">
|
|
370
387
|
<p class="ck-api-method"><span class="ck-chip ck-chip--soft">POST</span> /api/v1/provider_credentials</p>
|
|
371
|
-
<p class="ck-meta-copy">Create a provider credential.</p>
|
|
372
|
-
<p class="ck-api-params"><strong>Required:</strong> <code>provider</code> (openai, anthropic, ollama, openrouter
|
|
388
|
+
<p class="ck-meta-copy">Create a provider credential. Each provider may have only one credential; creating a second for an already-configured provider returns 422.</p>
|
|
389
|
+
<p class="ck-api-params"><strong>Required:</strong> <code>provider</code> (openai, anthropic, ollama, openrouter, azure_foundry) <strong>Optional:</strong> <code>api_key</code> (not enforced by the API, but required in practice for hosted providers; local providers like ollama can omit it), <code>api_endpoint</code> (required when <code>provider</code> is <code>azure_foundry</code>), <code>api_version</code></p>
|
|
390
|
+
<%= render "completion_kit/api_reference/example", base_url: base_url, token: token, real_token: real_token, cmd: "curl -X POST #{base_url}/api/v1/provider_credentials \\\n -H \"Authorization: Bearer #{token}\" \\\n -H \"Content-Type: application/json\" \\\n -d '{\"provider\": \"openai\", \"api_key\": \"sk-...\"}'" %>
|
|
391
|
+
<%= render "completion_kit/api_reference/example", base_url: base_url, token: token, real_token: real_token, cmd: "curl -X POST #{base_url}/api/v1/provider_credentials \\\n -H \"Authorization: Bearer #{token}\" \\\n -H \"Content-Type: application/json\" \\\n -d '{\"provider\": \"azure_foundry\", \"api_key\": \"...\", \"api_endpoint\": \"https://my-resource.openai.azure.com\", \"api_version\": \"2024-10-21\"}'" %>
|
|
373
392
|
</div>
|
|
374
393
|
<div class="ck-api-endpoint">
|
|
375
394
|
<p class="ck-api-method"><span class="ck-chip ck-chip--soft">GET</span> <span class="ck-chip ck-chip--soft">PATCH</span> <span class="ck-chip" style="color: var(--ck-danger);">DELETE</span> /api/v1/provider_credentials/:id</p>
|
|
376
|
-
<p class="ck-meta-copy">Get, update, or delete a provider credential.</p>
|
|
395
|
+
<p class="ck-meta-copy">Get, update, or delete a provider credential. DELETE returns 422 (not 204) when the provider is still in use by prompts or judge runs; remove those references first.</p>
|
|
396
|
+
<%= render "completion_kit/api_reference/example", base_url: base_url, token: token, real_token: real_token, cmd: "curl -X PATCH #{base_url}/api/v1/provider_credentials/1 \\\n -H \"Authorization: Bearer #{token}\" \\\n -H \"Content-Type: application/json\" \\\n -d '{\"api_key\": \"sk-new...\"}'" %>
|
|
377
397
|
</div>
|
|
378
398
|
<%= render "completion_kit/api_reference/resource_list", title: "Your providers",
|
|
379
399
|
items: provider_credentials.map { |pc|
|
|
@@ -384,12 +404,12 @@
|
|
|
384
404
|
|
|
385
405
|
<div class="ck-api-tabs__panel">
|
|
386
406
|
<h2 class="ck-section-title">Imports</h2>
|
|
387
|
-
<p class="ck-copy">Bring an existing <a href="https://www.promptfoo.dev" class="ck-link">promptfoo</a> config into CompletionKit in one call. Prompts, the test dataset, assert-based metrics
|
|
407
|
+
<p class="ck-copy">Bring an existing <a href="https://www.promptfoo.dev" class="ck-link">promptfoo</a> config into CompletionKit in one call. Prompts, the test dataset, and assert-based metrics are created where they map cleanly and skipped with a reason where they don't. Providers are matched against your already-configured credentials; any referenced in the config but not yet configured are reported so you can add them.</p>
|
|
388
408
|
<div class="ck-api-endpoint">
|
|
389
409
|
<p class="ck-api-method"><span class="ck-chip ck-chip--soft">POST</span> /api/v1/imports/promptfoo</p>
|
|
390
410
|
<p class="ck-meta-copy">Import a promptfooconfig.yaml. Send the YAML as a <code>config</code> param, or POST the raw YAML as the request body. Returns 201 with a mapping summary, or 422 if the YAML cannot be parsed.</p>
|
|
391
411
|
<p class="ck-api-params"><strong>Request:</strong> <code>config</code> (the YAML text) or a raw YAML request body</p>
|
|
392
|
-
<p class="ck-api-params"><strong>Response 201:</strong> <code>prompts</code
|
|
412
|
+
<p class="ck-api-params"><strong>Response 201:</strong> <code>prompts</code> and <code>metrics</code> each list what was <code>created</code> and what was <code>skipped</code> (with a reason); <code>dataset</code> is <code>{created, rows, columns}</code> (or <code>{skipped}</code>); <code>providers</code> is <code>{matched, unmatched}</code> (provider strings from the config)</p>
|
|
393
413
|
<%= render "completion_kit/api_reference/example", base_url: base_url, token: token, real_token: real_token, cmd: "curl -X POST #{base_url}/api/v1/imports/promptfoo \\\n -H \"Authorization: Bearer #{token}\" \\\n -H \"Content-Type: application/x-yaml\" \\\n --data-binary @promptfooconfig.yaml" %>
|
|
394
414
|
</div>
|
|
395
415
|
</div>
|