completion-kit 0.29.3 → 0.30.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/LICENSE +4 -1
- data/README.md +94 -127
- data/app/controllers/completion_kit/api/v1/base_controller.rb +1 -0
- data/app/services/completion_kit/mcp_tools/agreements.rb +4 -2
- data/app/services/completion_kit/mcp_tools/base.rb +7 -1
- data/app/services/completion_kit/mcp_tools/datasets.rb +12 -6
- data/app/services/completion_kit/mcp_tools/imports.rb +2 -1
- data/app/services/completion_kit/mcp_tools/judges.rb +4 -2
- data/app/services/completion_kit/mcp_tools/metric_groups.rb +10 -5
- data/app/services/completion_kit/mcp_tools/metric_versions.rb +6 -3
- data/app/services/completion_kit/mcp_tools/metrics.rb +12 -6
- data/app/services/completion_kit/mcp_tools/prompts.rb +14 -7
- data/app/services/completion_kit/mcp_tools/provider_credentials.rb +10 -5
- data/app/services/completion_kit/mcp_tools/responses.rb +4 -6
- data/app/services/completion_kit/mcp_tools/runs.rb +18 -11
- data/app/services/completion_kit/mcp_tools/tags.rb +10 -5
- data/app/views/completion_kit/api_reference/_body.html.erb +3 -5
- data/app/views/completion_kit/provider_credentials/_form.html.erb +4 -2
- data/lib/completion_kit/version.rb +1 -1
- data/lib/completion_kit.rb +2 -1
- metadata +1 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 7a88e90a186d2b535eb032d4ae9ecbae34605c622cc015f605044e5aaf570598
|
|
4
|
+
data.tar.gz: bcf073d39956a5ccd1ac1ccf2e95a69eba4e08998228b2d3ef264c1708daa523
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 99a8354ff64bb6339c44a5074787292d112eba92c9e9ba133b9e8fb3ab7364535f3a32b8e5e8d012264aaffea3572482b753f7e1c136a8ee303d55ef5232d3b3
|
|
7
|
+
data.tar.gz: 6de209e19ec52332cfa510e8abf2e33298bd674ff8b1fba239039cc64215e620a59f30cf6745990c5da448e40f49aa84cdb722dba2afd2f7985788469fb97e09
|
data/LICENSE
CHANGED
|
@@ -12,7 +12,10 @@ Additional Use Grant: You may use the Licensed Work for any purpose,
|
|
|
12
12
|
hosted or managed service whose primary value is the
|
|
13
13
|
functionality of the Licensed Work itself.
|
|
14
14
|
|
|
15
|
-
Change Date:
|
|
15
|
+
Change Date: Three years from the date each version of the
|
|
16
|
+
Licensed Work is first publicly released.
|
|
17
|
+
Versions 0.3.0 through 0.29.3 keep the Change
|
|
18
|
+
Date of 2029-04-25 they were released with.
|
|
16
19
|
|
|
17
20
|
Change License: GNU General Public License (GPL) Version 3
|
|
18
21
|
|
data/README.md
CHANGED
|
@@ -10,27 +10,50 @@
|
|
|
10
10
|
|
|
11
11
|
Your prompts need tests too.
|
|
12
12
|
|
|
13
|
-
Run every prompt against real data. Score each output with an LLM judge
|
|
13
|
+
Run every prompt against real data. Score each output against criteria you define, with an LLM judge or a plain check. Change anything: the prompt, the model, the temperature, the dataset. Re-run and see exactly what got better and what broke. When the scores tell you something's off, CompletionKit suggests an improved prompt based on the judge's feedback on your runs. You inspect the diff, apply it as a new version, and check the improvement.
|
|
14
14
|
|
|
15
15
|
It's the difference between "this prompt seems to work" and "this prompt scores 4.3 out of 5 across 200 inputs, up from 3.8 last version."
|
|
16
16
|
|
|
17
17
|
**[Start on completionkit.com →](https://completionkit.com)** | **[RubyGems](https://rubygems.org/gems/completion-kit)**
|
|
18
18
|
|
|
19
|
-
|
|
19
|
+

|
|
20
20
|
|
|
21
|
-
|
|
21
|
+
## How it works
|
|
22
|
+
|
|
23
|
+
<p align="center">
|
|
24
|
+
<img src="https://raw.githubusercontent.com/homemade-software-inc/completion-kit/main/docs/diagrams/workflow.png" alt="CompletionKit workflow: a prompt and a dataset feed a run against a model, an LLM judge scores each output on your rubric, low scores drive an AI-suggested rewrite, and the new prompt version re-runs so you can compare" width="820" />
|
|
25
|
+
</p>
|
|
26
|
+
|
|
27
|
+
It's a loop. Each pass leaves you with a score you can compare against the last one.
|
|
28
|
+
|
|
29
|
+
1. **Create a prompt** with `{{variable}}` placeholders.
|
|
30
|
+
2. **Upload a dataset.** A CSV whose column headers match the variable names.
|
|
31
|
+
3. **Run it** against a model and score the outputs against your metrics.
|
|
32
|
+
4. **Iterate.** Change the prompt, the model, the temperature, or the dataset and re-run. CompletionKit versions your prompts, so you can always compare against earlier results.
|
|
33
|
+
5. **Get suggestions.** When scores drop, ask CompletionKit for an improved prompt. The suggestion comes from the judge's feedback on each response, not generic prompt advice. Inspect the diff and apply it as a new version.
|
|
34
|
+
|
|
35
|
+
## Concepts
|
|
36
|
+
|
|
37
|
+
- **Prompt.** A versioned template with `{{variable}}` placeholders. Editing a prompt that has already run creates a new version, so earlier results stay reproducible.
|
|
38
|
+
- **Dataset.** A CSV of real inputs. Each row becomes one test case.
|
|
39
|
+
- **Run.** One execution of a prompt against a dataset. It records every input (model, temperature, metrics) and stores every output and score.
|
|
40
|
+
- **Response.** The model's output for one dataset row, with its scores attached.
|
|
41
|
+
- **Metric.** What you score each response on. A judge metric has an instruction, evaluation steps and a 1 to 5 scale, and an LLM judge applies it. A check metric is a pass or fail test with no LLM involved: contains, does not contain, equals, regex, valid JSON, JSON field equals, length bounds, list overlap, number bounds, and number equals.
|
|
42
|
+
- **Metric group.** A set of metrics you can apply to a run together.
|
|
43
|
+
- **Tag.** A label for prompts, runs, metrics and datasets. Filter any list by tag (`?tag[]=...`).
|
|
44
|
+
- **Provider credential.** An API key for a model provider. It is encrypted at rest and never returned through the API.
|
|
22
45
|
|
|
23
46
|
## Three ways to run it
|
|
24
47
|
|
|
25
|
-
|
|
48
|
+
Every option has the same engine, UI, REST API and MCP server. The first two work with any stack: you run CompletionKit as a product and talk to it over HTTP and MCP, whatever language your app is written in. The third is for teams already on Rails.
|
|
26
49
|
|
|
27
|
-
### 1. Hosted
|
|
50
|
+
### 1. Hosted at [completionkit.com](https://completionkit.com) (recommended)
|
|
28
51
|
|
|
29
|
-
|
|
52
|
+
Sign up and start testing prompts. There's no Rails app to run, no migrations, no job worker and no servers to keep up. Your whole team shares a workspace. Plans are at [completionkit.com/pricing](https://completionkit.com/pricing).
|
|
30
53
|
|
|
31
|
-
### 2. Self-hosted
|
|
54
|
+
### 2. Self-hosted standalone app
|
|
32
55
|
|
|
33
|
-
Run it on your own
|
|
56
|
+
Run it on your own servers as a self-contained product. There's nothing to integrate and no Ruby to write: once it's up, you drive it through the web UI, the REST API and the MCP server. It needs Postgres and a host that can run a Rails app.
|
|
34
57
|
|
|
35
58
|
```bash
|
|
36
59
|
git clone https://github.com/homemade-software-inc/completion-kit.git
|
|
@@ -40,7 +63,7 @@ bin/rails completion_kit:install:migrations
|
|
|
40
63
|
bin/rails db:migrate
|
|
41
64
|
```
|
|
42
65
|
|
|
43
|
-
Run
|
|
66
|
+
Run a web server and a Solid Queue worker, in two terminals:
|
|
44
67
|
|
|
45
68
|
```bash
|
|
46
69
|
bin/rails server
|
|
@@ -50,11 +73,11 @@ bin/rails server
|
|
|
50
73
|
bin/jobs
|
|
51
74
|
```
|
|
52
75
|
|
|
53
|
-
Or with [foreman](https://github.com/ddollar/foreman)
|
|
76
|
+
Or run both in one terminal with [foreman](https://github.com/ddollar/foreman): `foreman start -f Procfile.dev`.
|
|
54
77
|
|
|
55
|
-
Visit `http://localhost:3000`. Add a provider credential
|
|
78
|
+
Visit `http://localhost:3000`. Add a provider credential under Settings, create a prompt, upload a CSV dataset, and run it. See [Deploying self-hosted](#deploying-self-hosted) when you're ready for production.
|
|
56
79
|
|
|
57
|
-
### 3.
|
|
80
|
+
### 3. Mounted in your Rails app
|
|
58
81
|
|
|
59
82
|
```ruby
|
|
60
83
|
gem "completion-kit"
|
|
@@ -65,22 +88,24 @@ bin/rails generate completion_kit:install
|
|
|
65
88
|
bin/rails db:migrate
|
|
66
89
|
```
|
|
67
90
|
|
|
68
|
-
The engine mounts at `/completion_kit`.
|
|
91
|
+
The engine mounts at `/completion_kit`. Runs work through Active Job (`CompletionKit::GenerateRowJob`, `CompletionKit::JudgeReviewJob`, `CompletionKit::RunCompletionCheckJob` and others), so your app needs a real job backend such as Solid Queue, Sidekiq or GoodJob. Don't use the `:async` adapter in production. It runs jobs inside Puma's thread pool with no durability or retries, and a long LLM call will hold up web requests.
|
|
92
|
+
|
|
93
|
+
#### Using your own layout
|
|
69
94
|
|
|
70
|
-
|
|
95
|
+
If your app renders the engine's pages inside its own layout, include the engine's stylesheet and JavaScript there:
|
|
71
96
|
|
|
72
97
|
```erb
|
|
73
98
|
<%= stylesheet_link_tag "completion_kit/application", media: "all" %>
|
|
74
99
|
<%= javascript_include_tag "completion_kit/application", defer: true, nonce: true %>
|
|
75
100
|
```
|
|
76
101
|
|
|
77
|
-
Without the JavaScript
|
|
102
|
+
Without the JavaScript, much of the UI quietly stops working: clickable rows, dialogs, copy buttons, the run form, live run status and local times among them.
|
|
78
103
|
|
|
79
|
-
|
|
104
|
+
#### Content Security Policy
|
|
80
105
|
|
|
81
|
-
|
|
106
|
+
The engine's pages have no inline scripts or inline event handlers, and the engine layout puts the request nonce on its script tags, so they work under a nonce-based `script-src`. The views still use inline `style` attributes, so keep `style-src` out of the nonce directives and allow `'unsafe-inline'` there for now.
|
|
82
107
|
|
|
83
|
-
|
|
108
|
+
Use a nonce that lasts the whole session. Turbo Drive re-creates scripts on each visit using the nonce from the `csp-nonce` meta tag, but the browser keeps enforcing the policy from the first page it loaded, so a fresh nonce per request blocks scripts after the first visit. The Rails default nonce, the session id, is empty on a new visitor's first response, so it blocks that first page's scripts. Keep a random nonce in the session instead, and skip API controllers so API and MCP calls don't get a session cookie:
|
|
84
109
|
|
|
85
110
|
```ruby
|
|
86
111
|
Rails.application.configure do
|
|
@@ -101,25 +126,25 @@ end
|
|
|
101
126
|
|
|
102
127
|
## Providers
|
|
103
128
|
|
|
104
|
-
CompletionKit
|
|
129
|
+
CompletionKit finds the available models from each provider's API.
|
|
105
130
|
|
|
106
131
|
| Provider | Env vars | What it covers |
|
|
107
132
|
|----------|----------|----------------|
|
|
108
|
-
| **OpenAI** | `OPENAI_API_KEY` | GPT-5, GPT-4.1, GPT-4o
|
|
133
|
+
| **OpenAI** | `OPENAI_API_KEY` | GPT-5, GPT-4.1, GPT-4o and the rest |
|
|
109
134
|
| **Anthropic** | `ANTHROPIC_API_KEY` | Claude Opus, Sonnet, Haiku |
|
|
110
|
-
| **Ollama
|
|
111
|
-
| **OpenRouter** | `OPENROUTER_API_KEY` | 100+ models from 30+ providers through one
|
|
112
|
-
| **Azure AI Foundry** |
|
|
135
|
+
| **Ollama or a local endpoint** | `OLLAMA_API_ENDPOINT` (default `http://localhost:11434/v1`), `OLLAMA_API_KEY` (optional) | Any model you've pulled with Ollama, or any OpenAI-compatible local server (vLLM, LM Studio, llama.cpp) |
|
|
136
|
+
| **OpenRouter** | `OPENROUTER_API_KEY` | 100+ models from 30+ providers through one key |
|
|
137
|
+
| **Azure AI Foundry** | None. Add it as a provider credential with `api_key` and `api_endpoint` (`api_version` is optional) | Azure OpenAI deployments and Foundry projects |
|
|
113
138
|
|
|
114
|
-
Set these as environment variables
|
|
139
|
+
Set these as environment variables, in the generated initializer, or as provider credentials under Settings in the web UI.
|
|
115
140
|
|
|
116
|
-
The Ollama key from `OLLAMA_API_KEY` or `config.ollama_api_key`
|
|
141
|
+
The Ollama key from `OLLAMA_API_KEY` or `config.ollama_api_key` is only ever sent to the Ollama endpoint you configured, or to `http://localhost:11434/v1` when you configured none and loopback endpoints are allowed. A provider credential without its own key borrows that key only while it points at that same endpoint. Point it anywhere else and CompletionKit sends no key, so store the key on the credential. A key stored on a credential always goes to that credential's endpoint. Endpoints match on scheme, host, port and path; letter case, a written-out default port, a trailing slash and a trailing `/v1` don't count. A blank stored key counts as no key.
|
|
117
142
|
|
|
118
|
-
|
|
143
|
+
Custom model endpoints and CSV imports by URL may reach the server's own loopback interface by default, so a local Ollama works out of the box. If the people using CompletionKit shouldn't reach services on the app server itself, set `config.allow_loopback_endpoints = false`. Private, link-local and reserved addresses are always refused.
|
|
119
144
|
|
|
120
|
-
|
|
145
|
+
### Encryption
|
|
121
146
|
|
|
122
|
-
Generate them:
|
|
147
|
+
Provider API keys are encrypted at rest with [Active Record encryption](https://guides.rubyonrails.org/active_record_encryption.html). The app won't boot in production until three encryption keys are set. Generate them:
|
|
123
148
|
|
|
124
149
|
```bash
|
|
125
150
|
bin/rails db:encryption:init
|
|
@@ -133,13 +158,13 @@ COMPLETION_KIT_ENCRYPTION_DETERMINISTIC_KEY=<generated value>
|
|
|
133
158
|
COMPLETION_KIT_ENCRYPTION_KEY_DERIVATION_SALT=<generated value>
|
|
134
159
|
```
|
|
135
160
|
|
|
136
|
-
|
|
161
|
+
You can put them in `config/credentials.yml.enc` under `active_record_encryption` instead. In development the standalone app falls back to built-in values, so you can skip this locally.
|
|
137
162
|
|
|
138
163
|
## Authentication
|
|
139
164
|
|
|
140
|
-
CompletionKit
|
|
165
|
+
CompletionKit needs authentication anywhere it's deployed. In development and test the routes are open, with a warning in the log. Every other environment returns 403 until you configure one of these two modes.
|
|
141
166
|
|
|
142
|
-
### Basic
|
|
167
|
+
### Basic auth
|
|
143
168
|
|
|
144
169
|
```ruby
|
|
145
170
|
CompletionKit.configure do |c|
|
|
@@ -148,7 +173,7 @@ CompletionKit.configure do |c|
|
|
|
148
173
|
end
|
|
149
174
|
```
|
|
150
175
|
|
|
151
|
-
###
|
|
176
|
+
### Your own auth (Devise and the like)
|
|
152
177
|
|
|
153
178
|
```ruby
|
|
154
179
|
CompletionKit.configure do |c|
|
|
@@ -156,11 +181,9 @@ CompletionKit.configure do |c|
|
|
|
156
181
|
end
|
|
157
182
|
```
|
|
158
183
|
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
### Reviewer name
|
|
184
|
+
### Reviewer names
|
|
162
185
|
|
|
163
|
-
When someone
|
|
186
|
+
When someone agrees or disagrees with a judge's score in the web UI, CompletionKit saves that verdict under a reviewer name. Each reviewer keeps one verdict per response and metric, so give each person their own name or their verdicts overwrite each other. Set `current_reviewer` to a callable that takes the controller and returns the signed-in person's name:
|
|
164
187
|
|
|
165
188
|
```ruby
|
|
166
189
|
CompletionKit.configure do |c|
|
|
@@ -169,7 +192,9 @@ CompletionKit.configure do |c|
|
|
|
169
192
|
end
|
|
170
193
|
```
|
|
171
194
|
|
|
172
|
-
|
|
195
|
+
It must return a String or nil. Anything else, such as a user record, raises `CompletionKit::ConfigurationError`. When `current_reviewer` is unset or returns a blank value, verdicts go under `username`, or `"operator"` if there's no username.
|
|
196
|
+
|
|
197
|
+
The web UI doesn't read the `X-Remote-User` header by itself. If a proxy in front of your app sets it, pass it through:
|
|
173
198
|
|
|
174
199
|
```ruby
|
|
175
200
|
CompletionKit.configure do |c|
|
|
@@ -177,71 +202,46 @@ CompletionKit.configure do |c|
|
|
|
177
202
|
end
|
|
178
203
|
```
|
|
179
204
|
|
|
180
|
-
Only
|
|
205
|
+
Only do this when a trusted proxy always overwrites that header. Otherwise any client can send it and claim any name.
|
|
181
206
|
|
|
182
207
|
The REST API and the MCP server take the reviewer name from their own `created_by` parameter.
|
|
183
208
|
|
|
184
|
-
##
|
|
209
|
+
## Limits
|
|
210
|
+
|
|
211
|
+
### Rate limits
|
|
185
212
|
|
|
186
|
-
The REST API, the MCP endpoint
|
|
213
|
+
The REST API, the MCP endpoint and the web UI are rate limited per IP, per minute. The defaults are generous:
|
|
187
214
|
|
|
188
215
|
```ruby
|
|
189
216
|
CompletionKit.configure do |c|
|
|
190
|
-
c.api_rate_limit = 120 # REST API
|
|
191
|
-
c.web_rate_limit = 300 # web UI, requests per minute
|
|
217
|
+
c.api_rate_limit = 120 # REST API and MCP, requests per minute
|
|
218
|
+
c.web_rate_limit = 300 # web UI, requests per minute
|
|
192
219
|
end
|
|
193
220
|
```
|
|
194
221
|
|
|
195
|
-
|
|
222
|
+
Limits are counted in `Rails.cache`. A shared store such as Solid Cache or Redis counts across all your app instances. A per-process store limits each instance on its own.
|
|
196
223
|
|
|
197
|
-
|
|
224
|
+
### Input limits
|
|
198
225
|
|
|
199
|
-
Uploads, prompt templates
|
|
226
|
+
Uploads, prompt templates and regex checks have limits, so one huge or hostile input can't tie up a worker:
|
|
200
227
|
|
|
201
228
|
```ruby
|
|
202
229
|
CompletionKit.configure do |c|
|
|
203
230
|
c.max_upload_bytes = 25 * 1024 * 1024 # largest dataset CSV or import file (default 25 MB)
|
|
204
231
|
c.max_template_bytes = 1024 * 1024 # largest prompt template (default 1 MB)
|
|
205
|
-
c.regex_timeout = 1.0 # seconds a regex check may spend
|
|
232
|
+
c.regex_timeout = 1.0 # seconds a regex check may spend on one row (default 1.0)
|
|
206
233
|
end
|
|
207
234
|
```
|
|
208
235
|
|
|
209
|
-
A prompt
|
|
210
|
-
|
|
211
|
-
The first line of a dataset's CSV is its header row. A dataset fails validation when its CSV is saved with a header row that is blank or longer than 64 KB. The 64 KB limit is fixed, not a setting.
|
|
236
|
+
A prompt over `max_template_bytes` fails validation when it's saved. Prompts saved before you lowered the limit keep working until their template changes.
|
|
212
237
|
|
|
213
|
-
A dataset
|
|
238
|
+
A dataset fails validation when its header row is blank or longer than 64 KB (a fixed limit), or when the CSV isn't valid UTF-8. Export spreadsheets as UTF-8 CSV.
|
|
214
239
|
|
|
215
|
-
A regex check that runs
|
|
216
|
-
|
|
217
|
-
## How it works
|
|
218
|
-
|
|
219
|
-
<p align="center">
|
|
220
|
-
<img src="https://raw.githubusercontent.com/homemade-software-inc/completion-kit/main/docs/diagrams/workflow.png" alt="CompletionKit workflow: a prompt and a dataset feed a run against a model, an LLM judge scores each output on your rubric, low scores drive an AI-suggested rewrite, and the new prompt version re-runs so you can compare" width="820" />
|
|
221
|
-
</p>
|
|
222
|
-
|
|
223
|
-
It's a loop. Each pass leaves you with a score you can compare against the last one.
|
|
224
|
-
|
|
225
|
-
1. **Create a prompt** with `{{variable}}` placeholders
|
|
226
|
-
2. **Upload a dataset.** A CSV where column headers match the variable names.
|
|
227
|
-
3. **Run it** against a model and score outputs with an LLM judge against criteria you define.
|
|
228
|
-
4. **Iterate.** Change the prompt, the model, the temperature, or the dataset and re-run. CompletionKit versions your prompts so you can always compare against previous results.
|
|
229
|
-
5. **Get suggestions.** When scores drop, ask CompletionKit for an AI-generated improvement. The suggestion is based on the judge's actual per-response feedback, not generic prompt-engineering advice. Inspect the diff and apply it as a new version.
|
|
230
|
-
|
|
231
|
-
## Concepts
|
|
232
|
-
|
|
233
|
-
- **Prompt.** A versioned template with `{{variable}}` placeholders. Editing a prompt that's already been run creates a new version, so earlier results stay reproducible.
|
|
234
|
-
- **Dataset.** A CSV of real inputs. Each row becomes one test case.
|
|
235
|
-
- **Run.** One execution of a prompt against a dataset. Captures every input (model, temperature, metrics) and stores all outputs and scores.
|
|
236
|
-
- **Response.** The model's output for one dataset row, with reviews attached.
|
|
237
|
-
- **Metric.** An evaluation dimension with a name, instruction, evaluation steps, and a 1-5 star scoring scale. The LLM judge uses this to score each response.
|
|
238
|
-
- **Metric Group.** A reusable group of metrics you can apply to a run as a set.
|
|
239
|
-
- **Tag.** A domain label you can attach to prompts, runs, metrics, and datasets. Auto-assigned from a 10-color palette. Filter any index page by tag (`?tag[]=...`).
|
|
240
|
-
- **Provider Credential.** An API key for a model provider. Encrypted at rest, never returned through the API.
|
|
240
|
+
A regex check that runs out of time fails that row and says so. The limit is `regex_timeout`, or your app's `Regexp.timeout` if that's shorter, so the engine never loosens a stricter limit you set. It sets the limit per pattern and never touches the process-wide `Regexp.timeout`. Set `regex_timeout` to at least `0.001` seconds, or to `nil` to rely on `Regexp.timeout` alone. Any other value, such as `0` or `Float::INFINITY`, raises an `ArgumentError` when the initializer runs.
|
|
241
241
|
|
|
242
242
|
## REST API
|
|
243
243
|
|
|
244
|
-
Every resource is
|
|
244
|
+
Every resource is available through a JSON API with a bearer token:
|
|
245
245
|
|
|
246
246
|
```ruby
|
|
247
247
|
CompletionKit.configure { |c| c.api_token = ENV["COMPLETION_KIT_API_TOKEN"] }
|
|
@@ -252,11 +252,11 @@ curl -H "Authorization: Bearer $TOKEN" \
|
|
|
252
252
|
http://localhost:3000/completion_kit/api/v1/prompts
|
|
253
253
|
```
|
|
254
254
|
|
|
255
|
-
Visit `/completion_kit/api_reference` in your running app for
|
|
255
|
+
Visit `/completion_kit/api_reference` in your running app for docs on every endpoint, with curl examples you can copy.
|
|
256
256
|
|
|
257
257
|
## MCP server
|
|
258
258
|
|
|
259
|
-
CompletionKit runs a [Model Context Protocol](https://modelcontextprotocol.io) server at `/completion_kit/mcp
|
|
259
|
+
CompletionKit runs a [Model Context Protocol](https://modelcontextprotocol.io) server at `/completion_kit/mcp`. It exposes every resource as tools that MCP clients such as Claude Code and Cursor can use directly:
|
|
260
260
|
|
|
261
261
|
```json
|
|
262
262
|
{
|
|
@@ -269,21 +269,21 @@ CompletionKit runs a [Model Context Protocol](https://modelcontextprotocol.io) s
|
|
|
269
269
|
}
|
|
270
270
|
```
|
|
271
271
|
|
|
272
|
-
The
|
|
272
|
+
The API reference page in the app has snippets you can paste into your MCP client's config.
|
|
273
273
|
|
|
274
274
|
## Deploying self-hosted
|
|
275
275
|
|
|
276
|
-
Any Rails
|
|
276
|
+
Any host that runs Rails will do: Fly, Render, Heroku, Docker and so on. Point it at Postgres with `DATABASE_URL`, set your provider env vars, and run `cd standalone && bin/rails db:migrate` on each deploy.
|
|
277
277
|
|
|
278
278
|
| Variable | Purpose | Default |
|
|
279
279
|
|----------|---------|---------|
|
|
280
|
-
| `COMPLETION_KIT_API_TOKEN` | Bearer token for REST API and MCP |
|
|
281
|
-
| `COMPLETION_KIT_USERNAME` | Web UI
|
|
282
|
-
| `COMPLETION_KIT_PASSWORD` | Web UI
|
|
280
|
+
| `COMPLETION_KIT_API_TOKEN` | Bearer token for the REST API and MCP | None, so the API is off |
|
|
281
|
+
| `COMPLETION_KIT_USERNAME` | Web UI username | `admin` |
|
|
282
|
+
| `COMPLETION_KIT_PASSWORD` | Web UI password | None, so it's open in development only |
|
|
283
283
|
|
|
284
|
-
You also need the three `COMPLETION_KIT_ENCRYPTION_*` keys from
|
|
284
|
+
You also need the three `COMPLETION_KIT_ENCRYPTION_*` keys from [Encryption](#encryption).
|
|
285
285
|
|
|
286
|
-
When
|
|
286
|
+
When a new gem version ships a migration, install it locally and commit it before you deploy:
|
|
287
287
|
|
|
288
288
|
```bash
|
|
289
289
|
cd standalone
|
|
@@ -294,13 +294,13 @@ git add db/migrate/ && git commit -m "install new engine migration"
|
|
|
294
294
|
|
|
295
295
|
### Docker
|
|
296
296
|
|
|
297
|
-
The standalone app ships a `Dockerfile`, so you can
|
|
297
|
+
The standalone app ships a `Dockerfile`, so you can run it without Ruby on the host. Build from the repository root, since the app needs the engine source next to it:
|
|
298
298
|
|
|
299
299
|
```bash
|
|
300
300
|
docker build -f standalone/Dockerfile -t completion-kit .
|
|
301
301
|
```
|
|
302
302
|
|
|
303
|
-
CompletionKit needs a Rails secret (`SECRET_KEY_BASE`) and three
|
|
303
|
+
CompletionKit needs a Rails secret (`SECRET_KEY_BASE`) and the three encryption keys. Without Ruby on the host you can't run `bin/rails db:encryption:init`, so generate them with `openssl` and write everything to an env file:
|
|
304
304
|
|
|
305
305
|
```bash
|
|
306
306
|
cat > completion-kit.env <<EOF
|
|
@@ -312,58 +312,25 @@ COMPLETION_KIT_ENCRYPTION_KEY_DERIVATION_SALT=$(openssl rand -hex 32)
|
|
|
312
312
|
EOF
|
|
313
313
|
```
|
|
314
314
|
|
|
315
|
-
|
|
315
|
+
Generate these **once** and keep them. If the encryption keys change, the provider keys already in the database can't be decrypted. Keep `completion-kit.env` out of version control and back it up somewhere safe.
|
|
316
316
|
|
|
317
|
-
Run the web process and a job worker from the same image, both
|
|
317
|
+
Run the web process and a job worker from the same image and the same env file, so both share the same secrets:
|
|
318
318
|
|
|
319
319
|
```bash
|
|
320
320
|
docker run -d -p 3000:3000 --env-file completion-kit.env completion-kit
|
|
321
321
|
docker run -d --env-file completion-kit.env completion-kit ./bin/jobs
|
|
322
322
|
```
|
|
323
323
|
|
|
324
|
-
|
|
325
|
-
|
|
326
|
-
## Multi-tenant host apps (advanced)
|
|
327
|
-
|
|
328
|
-
For hosts that mount CompletionKit in a multi-tenant app, two optional hooks scope engine records per tenant without forking the engine:
|
|
329
|
-
|
|
330
|
-
```ruby
|
|
331
|
-
CompletionKit.configure do |config|
|
|
332
|
-
config.tenant_scope = -> {
|
|
333
|
-
org = Current.organization&.id
|
|
334
|
-
org ? where(organization_id: org) : where("1=0")
|
|
335
|
-
}
|
|
336
|
-
config.tenant_scope_columns = [:organization_id]
|
|
337
|
-
end
|
|
338
|
-
```
|
|
339
|
-
|
|
340
|
-
`tenant_scope` runs as each engine model's `default_scope` (use `unscoped` to bypass). `tenant_scope_columns` is appended to every engine uniqueness validation. Adding the tenant columns and composite unique indexes lives in your host migrations. Both defaults (`nil`, `[]`) are no-ops. The `completion_kit:mark_interrupted_runs_failed` rake task bypasses `tenant_scope` and marks in-flight runs in every tenant, so a host that also enforces tenancy in the database, for example with row-level security, must run it as a database role that can see every tenant's rows.
|
|
341
|
-
|
|
342
|
-
Custom model endpoints and CSV imports by URL may reach the server's own loopback interface by default, so a local Ollama works out of the box. If the people using the engine should not reach services on the app server itself, set `config.allow_loopback_endpoints = false`. Private, link-local and reserved addresses are always refused.
|
|
343
|
-
|
|
344
|
-
One hook lets a host apply its own display rule everywhere the engine lists, counts, or traverses runs, without overriding controllers or views:
|
|
345
|
-
|
|
346
|
-
```ruby
|
|
347
|
-
CompletionKit.configure do |config|
|
|
348
|
-
config.runs_display_scope = -> { where(created_at: 90.days.ago..) }
|
|
349
|
-
config.runs_display_footer_partial = "runs/display_notice"
|
|
350
|
-
end
|
|
351
|
-
```
|
|
352
|
-
|
|
353
|
-
`runs_display_scope` is a callable evaluated against a `Run` relation, in the same bare-`where` style as `tenant_scope` (it runs via `instance_exec`, so write it zero-arg with the relation as `self`, like a Rails `scope` lambda). It must return a relation: a callable that returns `nil` or anything non-chainable raises when a list renders. The engine applies it through `Run.display_scoped` at every run list and count it owns (the runs index, prompt and dataset show pages, the compare picker, new-run tag defaults, the v1 API index and its `X-Total-Count`, the MCP `runs_list` tool, the dashboard and API reference recent-runs panels, the dashboard "Runs" stat-card count, and provider-credential usage stats) and through `Run.visible_run_ids` for child records that traverse runs (the metric trust-panel sample and the agreement examples shown on a metric page). Use it for a list-only display rule rather than a global `default_scope`, which would null `Run` associations everywhere they are traversed.
|
|
354
|
-
|
|
355
|
-
Deliberately exempt, because they must still see every run: id-addressed single-run lookups (`runs#show`, the MCP `runs_get` tool, the v1 API show), delete-confirmation cascade counts, the auto-generated run-name counter, the dashboard's stat-card gate (it keys off the unscoped run total so a workspace whose runs the display scope excludes entirely still sees its activity cards rather than a blank dashboard), and the judge few-shot seeding that learns from corrected examples even on runs the display scope excludes.
|
|
356
|
-
|
|
357
|
-
`runs_display_footer_partial` names a partial rendered below the runs list on the dashboard, the index, and the prompt and dataset show pages; it receives the shown runs as a `runs` local. When the display scope excludes every run one of those surfaces would list, the surface says "No runs to show." and the footer partial renders beneath it. The engine cannot know why your scope excludes runs, so use the partial to explain your display rule: your host owns that rule in `runs_display_scope` and is the only side that can describe or count what it left out. Both default to `nil` (no-ops), leaving standalone behaviour unchanged.
|
|
324
|
+
The web container runs `db:prepare` on boot, so migrations apply on first start and on every deploy.
|
|
358
325
|
|
|
359
326
|
## Contributing
|
|
360
327
|
|
|
361
|
-
See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup, testing
|
|
328
|
+
See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup, testing and pull request guidelines.
|
|
362
329
|
|
|
363
330
|
## License
|
|
364
331
|
|
|
365
|
-
CompletionKit 0.3.0 and later are licensed under the [Business Source License 1.1](LICENSE). You may use CompletionKit
|
|
332
|
+
CompletionKit 0.3.0 and later are licensed under the [Business Source License 1.1](LICENSE). You may use CompletionKit for any purpose, including production, except offering it (or a derivative) to others as a hosted or managed service whose main value is CompletionKit itself. Each version becomes GPL-3 three years after it's released. Versions 0.3.0 through 0.29.3 shipped with a fixed date and become GPL-3 on April 25, 2029.
|
|
366
333
|
|
|
367
334
|
CompletionKit 0.2.x and earlier remain available under the [MIT License](https://github.com/homemade-software-inc/completion-kit/blob/v0.2.0/MIT-LICENSE).
|
|
368
335
|
|
|
369
|
-
For
|
|
336
|
+
For other licensing, contact hello@homemade.software.
|
|
@@ -3,6 +3,7 @@ module CompletionKit
|
|
|
3
3
|
module V1
|
|
4
4
|
class BaseController < ActionController::API
|
|
5
5
|
rate_limit to: CompletionKit.config.api_rate_limit, within: 1.minute,
|
|
6
|
+
by: -> { CompletionKit.config.api_rate_limit_key.call(request) },
|
|
6
7
|
with: -> { render_error("Rate limit exceeded", status: :too_many_requests) }
|
|
7
8
|
before_action :authenticate_api!
|
|
8
9
|
rescue_from ConfigurationError, with: :render_configuration_error
|
|
@@ -5,7 +5,8 @@ module CompletionKit
|
|
|
5
5
|
|
|
6
6
|
TOOLS = {
|
|
7
7
|
"agreements_list" => {
|
|
8
|
-
description: "List
|
|
8
|
+
description: "List human reviews of the LLM judge's scores: each agree, disagree or borderline verdict, with any corrected score and note. Filter by run_id, response_id, metric_id, or created_by. Use it to see where people disagreed with the judge before improving a metric's rubric. It returns each verdict, oldest first, with its run, response, metric and metric version ids, who recorded it and when.",
|
|
9
|
+
annotations: Base::READ_ONLY,
|
|
9
10
|
inputSchema: {
|
|
10
11
|
type: "object",
|
|
11
12
|
properties: {
|
|
@@ -19,7 +20,8 @@ module CompletionKit
|
|
|
19
20
|
handler: :list
|
|
20
21
|
},
|
|
21
22
|
"agreements_create" => {
|
|
22
|
-
description: "
|
|
23
|
+
description: "Record whether a person agrees with the LLM judge's score on one response for one metric, so you can measure how far to trust the judge. The response must belong to the given run. Verdict is one of agree, disagree, borderline. corrected_score (1..5) is required when verdict is 'disagree'. This upserts an agreement for (run, response, metric, created_by): a new verdict with the same created_by (default \"mcp\") replaces the earlier one, including its corrected score and note, and leaving either out clears it. The verdict is filed under the metric's current judge version, which is the version judges_compare counts it for, even when an earlier version scored that response. It works only for llm_judge metrics, because checks have nothing to calibrate. It returns the saved verdict.",
|
|
24
|
+
annotations: Base::DESTRUCTIVE,
|
|
23
25
|
inputSchema: {
|
|
24
26
|
type: "object",
|
|
25
27
|
properties: {
|
|
@@ -4,6 +4,12 @@ module CompletionKit
|
|
|
4
4
|
DEFAULT_PAGE_LIMIT = 50
|
|
5
5
|
MAX_PAGE_LIMIT = 500
|
|
6
6
|
|
|
7
|
+
READ_ONLY = {readOnlyHint: true, destructiveHint: false, openWorldHint: false}.freeze
|
|
8
|
+
ADDITIVE = {readOnlyHint: false, destructiveHint: false, openWorldHint: false}.freeze
|
|
9
|
+
DESTRUCTIVE = {readOnlyHint: false, destructiveHint: true, openWorldHint: false}.freeze
|
|
10
|
+
OPEN_WORLD_ADDITIVE = {readOnlyHint: false, destructiveHint: false, openWorldHint: true}.freeze
|
|
11
|
+
OPEN_WORLD_DESTRUCTIVE = {readOnlyHint: false, destructiveHint: true, openWorldHint: true}.freeze
|
|
12
|
+
|
|
7
13
|
def page_bounds(args)
|
|
8
14
|
limit = args["limit"].to_i
|
|
9
15
|
limit = DEFAULT_PAGE_LIMIT if limit <= 0
|
|
@@ -14,7 +20,7 @@ module CompletionKit
|
|
|
14
20
|
end
|
|
15
21
|
|
|
16
22
|
def definitions
|
|
17
|
-
self::TOOLS.map { |name, config| {name: name, description: config[:description], inputSchema: config[:inputSchema]} }
|
|
23
|
+
self::TOOLS.map { |name, config| {name: name, description: config[:description], inputSchema: config[:inputSchema], annotations: config[:annotations]} }
|
|
18
24
|
end
|
|
19
25
|
|
|
20
26
|
def call(name, arguments)
|
|
@@ -9,17 +9,20 @@ module CompletionKit
|
|
|
9
9
|
|
|
10
10
|
TOOLS = {
|
|
11
11
|
"datasets_list" => {
|
|
12
|
-
description: "List
|
|
12
|
+
description: "List every test dataset in your CompletionKit account, newest first. Returns each dataset's id, name, full CSV data, tags and timestamps, so the result can be large when your datasets are big. Use it to find a dataset to test a prompt against or to score AI responses on. To read one dataset, use datasets_get.",
|
|
13
|
+
annotations: Base::READ_ONLY,
|
|
13
14
|
inputSchema: {type: "object", properties: {}, required: []},
|
|
14
15
|
handler: :list
|
|
15
16
|
},
|
|
16
17
|
"datasets_get" => {
|
|
17
|
-
description: "Get
|
|
18
|
+
description: "Get one test dataset by its ID. Returns the dataset's name, its full CSV data (header row first, then one test case per row), its tags and its timestamps. Use it to check a dataset's columns and rows before you test a prompt or start a run against it.",
|
|
19
|
+
annotations: Base::READ_ONLY,
|
|
18
20
|
inputSchema: {type: "object", properties: {id: {type: "integer"}}, required: ["id"]},
|
|
19
21
|
handler: :get
|
|
20
22
|
},
|
|
21
23
|
"datasets_create" => {
|
|
22
|
-
description: "Create a dataset
|
|
24
|
+
description: "Create a test dataset from CSV text included in the call. The first row is the header, and each later row is one test case. Every column is available to the prompt as a variable. Two column names are recognized specially: \"expected_output\" is each row's answer key (ground truth) given to the judge and to checks that compare against the row's expected value, and \"actual_output\" is a pre-made output to score in a prompt-less run. Both are overridable per run (expected_column / output_column). Optional tag_names labels the dataset. Tag names are lowercased and may contain only unaccented letters, numbers, spaces, hyphens and underscores, up to 64 characters, and tags that don't exist yet are created. Returns the new dataset. The CSV is rejected if it is empty, is not valid UTF-8, has a blank header row or a header row over 64 KB, or is larger than the upload limit (25 MB by default). If the CSV is too big to put in the call, use datasets_create_from_url, which downloads files up to 10MB.",
|
|
25
|
+
annotations: Base::ADDITIVE,
|
|
23
26
|
inputSchema: {
|
|
24
27
|
type: "object",
|
|
25
28
|
properties: {name: {type: "string"}, csv_data: {type: "string"}, tag_names: {type: "array", items: {type: "string"}}},
|
|
@@ -28,7 +31,8 @@ module CompletionKit
|
|
|
28
31
|
handler: :create
|
|
29
32
|
},
|
|
30
33
|
"datasets_update" => {
|
|
31
|
-
description: "
|
|
34
|
+
description: "Change a test dataset's name, CSV data or tags. Passing csv_data replaces the whole CSV, and the dataset keeps no copy of the old version. Passing tag_names replaces all of the dataset's tags. Responses from runs that already ran keep the row values they used, but any run started after the change reads the new rows. If you remove or rename a column that a run needs (a variable in its prompt, its expected_column or its output column), that run cannot be started, finished, regraded, retried or rerun until the column is back. To add rows, read the dataset with datasets_get first and send back the full CSV. Returns the updated dataset. If tag_names is rejected, the name and CSV changes in the same call are still saved.",
|
|
35
|
+
annotations: Base::DESTRUCTIVE,
|
|
32
36
|
inputSchema: {
|
|
33
37
|
type: "object",
|
|
34
38
|
properties: {id: {type: "integer"}, name: {type: "string"}, csv_data: {type: "string"}, tag_names: {type: "array", items: {type: "string"}}},
|
|
@@ -37,12 +41,14 @@ module CompletionKit
|
|
|
37
41
|
handler: :update
|
|
38
42
|
},
|
|
39
43
|
"datasets_delete" => {
|
|
40
|
-
description: "
|
|
44
|
+
description: "Permanently delete a test dataset. This also deletes every run that used it, with those runs' responses, scores, suggestions and human agreement labels. Prompts and metrics are kept. This cannot be undone, so confirm with the user before calling it. Returns a short confirmation.",
|
|
45
|
+
annotations: Base::DESTRUCTIVE,
|
|
41
46
|
inputSchema: {type: "object", properties: {id: {type: "integer"}}, required: ["id"]},
|
|
42
47
|
handler: :delete
|
|
43
48
|
},
|
|
44
49
|
"datasets_create_from_url" => {
|
|
45
|
-
description: "Create a dataset by downloading CSV from a URL instead of
|
|
50
|
+
description: "Create a test dataset by downloading a CSV file from a public http(s) URL instead of pasting it into the call. Use this for large datasets: the server fetches the CSV directly, so the data never has to pass through the tool-call arguments. The URL is SSRF-checked, so it cannot point at private network addresses. Redirects are not followed and return an error, and files over 10MB are rejected. The first row is the header. The \"expected_output\" (answer key) and \"actual_output\" (pre-made output) columns are recognized specially, and both are overridable per run. Optional tag_names labels the dataset. Returns the new dataset, or an error that says why the URL, the download or the CSV was rejected.",
|
|
51
|
+
annotations: Base::OPEN_WORLD_ADDITIVE,
|
|
46
52
|
inputSchema: {
|
|
47
53
|
type: "object",
|
|
48
54
|
properties: {
|
|
@@ -5,7 +5,8 @@ module CompletionKit
|
|
|
5
5
|
|
|
6
6
|
TOOLS = {
|
|
7
7
|
"promptfoo_import" => {
|
|
8
|
-
description: "Import a promptfooconfig.yaml.
|
|
8
|
+
description: "Import a promptfoo config (the full promptfooconfig.yaml text) so you can test its prompts and score their responses in CompletionKit. It creates a new prompt for each inline prompt template, one dataset from the tests' vars, and metrics from the assert blocks in tests and defaultTest, with identical asserts imported once. llm-rubric, g-eval, model-graded-closedqa and factuality become LLM judge metrics. contains, icontains, not-contains, not-icontains, equals, regex and is-json become deterministic check metrics. Prompts use the model named by the first provider, or gpt-4o if none is given, and are created even if no saved provider key serves that model. Prompts given as file:// references are skipped, so paste the file's contents instead. It only adds new records, never changes existing ones and does not start a run. Providers are only matched by name against your saved provider keys and reported as matched or unmatched. Any apiKey in the config is ignored, so remove keys before pasting. Returns a summary of what was created and of the prompts and assert types it skipped, with the reason for each. Assert options such as threshold, weight and provider, defaultTest vars, and asserts with no type are dropped without being listed in the summary. If a prompt or the dataset cannot be saved, the whole import is rolled back and nothing is created.",
|
|
9
|
+
annotations: Base::ADDITIVE,
|
|
9
10
|
inputSchema: {
|
|
10
11
|
type: "object",
|
|
11
12
|
properties: {
|
|
@@ -5,7 +5,8 @@ module CompletionKit
|
|
|
5
5
|
|
|
6
6
|
TOOLS = {
|
|
7
7
|
"judges_replay" => {
|
|
8
|
-
description: "
|
|
8
|
+
description: "Check an LLM judge against outputs you already have: set up a scoring run that grades a dataset column of existing outputs with one metric's current judge and the judge_model you name (wraps runs_create with prompt_id omitted and output_column supplied, defaulting to actual_output). This only sets up the run; call runs_generate to actually re-judge the outputs so you can compare against human verdicts. It returns the new run.",
|
|
9
|
+
annotations: Base::ADDITIVE,
|
|
9
10
|
inputSchema: {
|
|
10
11
|
type: "object",
|
|
11
12
|
properties: {
|
|
@@ -20,7 +21,8 @@ module CompletionKit
|
|
|
20
21
|
handler: :replay
|
|
21
22
|
},
|
|
22
23
|
"judges_compare" => {
|
|
23
|
-
description: "Compare two versions of one metric's
|
|
24
|
+
description: "Compare two versions of one metric's LLM judge to see which agrees more with human reviewers. It computes stats from the agreement verdicts already recorded under each version and saves nothing; verdicts on runs older than the account's history window are not counted. It requires metric_id, metric_version_a_id, and metric_version_b_id (both versions must belong to that metric). It returns, for each version, the number of verdicts, the agreement rate with its confidence range, borderline rate, mean absolute error and kappa, plus the change from A to B and a recommendation. The recommendation is need_more_data under 30 verdicts combined, recommend when B's agreement rate beats A's by more than 3 points, hold when it trails A's by more than 3 points, and no_change otherwise or when one version has no verdicts. It is unavailable for check metrics.",
|
|
25
|
+
annotations: Base::READ_ONLY,
|
|
24
26
|
inputSchema: {
|
|
25
27
|
type: "object",
|
|
26
28
|
properties: {
|
|
@@ -5,17 +5,20 @@ module CompletionKit
|
|
|
5
5
|
|
|
6
6
|
TOOLS = {
|
|
7
7
|
"metric_groups_list" => {
|
|
8
|
-
description: "List
|
|
8
|
+
description: "List every metric group in your workspace, newest first. A metric group is a saved, ordered set of metrics that you can attach to a run in one step, so runs are scored the same way each time. Returns each group's id, name, description, metric_ids, tags and timestamps. The full list comes back with no paging. Use it to find a metric_group_id for runs_create or runs_update.",
|
|
9
|
+
annotations: Base::READ_ONLY,
|
|
9
10
|
inputSchema: {type: "object", properties: {}, required: []},
|
|
10
11
|
handler: :list
|
|
11
12
|
},
|
|
12
13
|
"metric_groups_get" => {
|
|
13
|
-
description: "Get
|
|
14
|
+
description: "Get one metric group by id. Returns its id, name, description, the ids of its metrics in order, tags and timestamps. Use metrics_get on each id to see how each metric scores AI responses.",
|
|
15
|
+
annotations: Base::READ_ONLY,
|
|
14
16
|
inputSchema: {type: "object", properties: {id: {type: "integer"}}, required: ["id"]},
|
|
15
17
|
handler: :get
|
|
16
18
|
},
|
|
17
19
|
"metric_groups_create" => {
|
|
18
|
-
description: "Create a metric group",
|
|
20
|
+
description: "Create a metric group: a named, ordered set of metrics that you can attach to a run in one step with metric_group_id, so every run of a prompt is scored the same way. Send name, and optionally description, metric_ids in the order you want them and tag_names (any new tags are created). Each id must be a metric in this workspace, or the whole call fails. A run copies the group's metrics when you attach the group, so later changes to the group don't reach that run. Returns the saved group.",
|
|
21
|
+
annotations: Base::ADDITIVE,
|
|
19
22
|
inputSchema: {
|
|
20
23
|
type: "object",
|
|
21
24
|
properties: {
|
|
@@ -28,7 +31,8 @@ module CompletionKit
|
|
|
28
31
|
handler: :create
|
|
29
32
|
},
|
|
30
33
|
"metric_groups_update" => {
|
|
31
|
-
description: "
|
|
34
|
+
description: "Change a metric group's name, description, metrics or tags. This overwrites the group in place. metric_ids replaces the group's whole metric list in the order given, and an empty list removes every metric. Each id must be a metric in this workspace. Leave metric_ids out to keep the current list. tag_names replaces the whole tag set. Runs already created keep the metrics they had. Returns the updated group.",
|
|
35
|
+
annotations: Base::DESTRUCTIVE,
|
|
32
36
|
inputSchema: {
|
|
33
37
|
type: "object",
|
|
34
38
|
properties: {
|
|
@@ -41,7 +45,8 @@ module CompletionKit
|
|
|
41
45
|
handler: :update
|
|
42
46
|
},
|
|
43
47
|
"metric_groups_delete" => {
|
|
44
|
-
description: "
|
|
48
|
+
description: "Permanently delete a metric group. Its tags are unlinked, though the tags themselves stay. The metrics in it, and any runs scored with it, stay as they are. This cannot be undone. Returns a confirmation message.",
|
|
49
|
+
annotations: Base::DESTRUCTIVE,
|
|
45
50
|
inputSchema: {type: "object", properties: {id: {type: "integer"}}, required: ["id"]},
|
|
46
51
|
handler: :delete
|
|
47
52
|
}
|
|
@@ -5,7 +5,8 @@ module CompletionKit
|
|
|
5
5
|
|
|
6
6
|
TOOLS = {
|
|
7
7
|
"metric_versions_list" => {
|
|
8
|
-
description: "List every
|
|
8
|
+
description: "List every version of one metric's judge or check, drafts and published, newest first. Use it to see how a judge changed over time, find drafts suggested by metrics_suggest_variants or saved from edits in the web app, or pick an older version to restore with metric_versions_publish. Each row carries id, version_number, state (draft or published), source, the current flag marking the live version, and timestamps, plus that version's instruction and rubric bands (judges) or check_config (checks). source is edit or suggestion for versions made those ways, and null for a version saved automatically the first time the metric scored a response or got a verdict. The current version can differ from the metric's live definition if metrics_update changed the metric after that version was saved.",
|
|
9
|
+
annotations: Base::READ_ONLY,
|
|
9
10
|
inputSchema: {
|
|
10
11
|
type: "object",
|
|
11
12
|
properties: {
|
|
@@ -16,7 +17,8 @@ module CompletionKit
|
|
|
16
17
|
handler: :list
|
|
17
18
|
},
|
|
18
19
|
"metric_versions_publish" => {
|
|
19
|
-
description: "
|
|
20
|
+
description: "Make a metric version the live judge or check that scores new AI responses. It works both to publish a draft and to switch back to an older published version. In one step it marks this version current, marks every other version of the metric not current, and writes this version's instruction and rubric bands (or check_config, for a check) onto the metric, replacing what the metric had. Responses already scored keep their scores. Rescore a run with runs_regrade if you want new ones. Human verdicts stay with the version they were given on. Other published versions stay saved, so you can switch back by publishing one, but a live definition that was never saved as a version is lost. A check version with an invalid check_config is refused. Returns the published version.",
|
|
21
|
+
annotations: Base::DESTRUCTIVE,
|
|
20
22
|
inputSchema: {
|
|
21
23
|
type: "object",
|
|
22
24
|
properties: {
|
|
@@ -27,7 +29,8 @@ module CompletionKit
|
|
|
27
29
|
handler: :publish
|
|
28
30
|
},
|
|
29
31
|
"metric_versions_dismiss" => {
|
|
30
|
-
description: "
|
|
32
|
+
description: "Permanently delete a draft version of a metric, whether it came from an edit in the web app or from metrics_suggest_variants. Use it to throw away a suggested judge rewrite you don't want. Published versions are refused. To stop using a published version, publish a different one as current instead. This cannot be undone. Returns the deleted version's id with destroyed: true.",
|
|
33
|
+
annotations: Base::DESTRUCTIVE,
|
|
31
34
|
inputSchema: {
|
|
32
35
|
type: "object",
|
|
33
36
|
properties: {
|
|
@@ -44,17 +44,20 @@ module CompletionKit
|
|
|
44
44
|
|
|
45
45
|
TOOLS = {
|
|
46
46
|
"metrics_list" => {
|
|
47
|
-
description: "List
|
|
47
|
+
description: "List every metric in your workspace, newest first. A metric is how CompletionKit scores AI responses. It is either an LLM judge, which grades each response from 1 to 5 (5 is best) against an instruction and rubric, or a check, which scores a response by an exact rule such as contains, equals, regex, valid JSON, a JSON path value, length or number bounds, or list overlap. Returns each metric's id, name, key, type, tags and timestamps, plus its instruction and rubric bands (judges) or its check_config (checks). The full list comes back with no paging. Use it to find a metric id before you score a run, edit a judge or build a metric group.",
|
|
48
|
+
annotations: Base::READ_ONLY,
|
|
48
49
|
inputSchema: {type: "object", properties: {}, required: []},
|
|
49
50
|
handler: :list
|
|
50
51
|
},
|
|
51
52
|
"metrics_get" => {
|
|
52
|
-
description: "Get
|
|
53
|
+
description: "Get one metric by id. Returns its id, name, key, type, tags and timestamps, plus its live judge instruction and 1 to 5 rubric bands (for an LLM judge) or its check_config (for a check). Use it to read exactly how a metric scores AI responses before you change it or explain a score. To see its saved versions and drafts, use metric_versions_list.",
|
|
54
|
+
annotations: Base::READ_ONLY,
|
|
53
55
|
inputSchema: {type: "object", properties: {id: {type: "integer"}}, required: ["id"]},
|
|
54
56
|
handler: :get
|
|
55
57
|
},
|
|
56
58
|
"metrics_create" => {
|
|
57
|
-
description: "Create a metric with
|
|
59
|
+
description: "Create a metric that scores AI responses. By default it is an LLM judge. Send name and instruction (what a good response looks like), and optionally rubric_bands describing 1 to 5 stars, where 5 is best. Any band you leave out gets a default description. For an exact rule instead of a judge, see the check details at the end. The metric's name must be unique in the workspace, ignoring case, spaces and most punctuation, and a renamed metric keeps its original name reserved. A clash fails with the error \"Key has already been taken\". tag_names adds tags and creates any that don't exist yet. Saving a metric scores nothing by itself. Attach it to a run with runs_create or runs_update, then score responses with runs_generate or runs_regrade. Returns the saved metric. #{CHECK_CONFIG_HINT}",
|
|
60
|
+
annotations: Base::ADDITIVE,
|
|
58
61
|
inputSchema: {
|
|
59
62
|
type: "object",
|
|
60
63
|
properties: {
|
|
@@ -69,7 +72,8 @@ module CompletionKit
|
|
|
69
72
|
handler: :create
|
|
70
73
|
},
|
|
71
74
|
"metrics_update" => {
|
|
72
|
-
description: "
|
|
75
|
+
description: "Change a metric's name, judge instruction, rubric bands, check_config, type or tags. This overwrites the metric in place, and the new definition scores every response graded after it. It saves no draft or new version. If the metric already has a live version, scores and verdicts given after the change are still filed under that version, so judges_compare can't tell the old and new definitions apart. Responses already scored keep their old scores until you rescore the run with runs_regrade. Nothing tests the change before it goes live. Fields you leave out stay as they are, but a field you send replaces that whole field: rubric_bands resets any band you leave out to its default description, not its old one, and check_config replaces the whole config. tag_names replaces the whole tag set. A rename keeps the metric's original key, so its old name stays reserved. metric_type cannot change once the metric is in a run, has scores or has versions. Returns the updated metric. #{CHECK_CONFIG_HINT}",
|
|
76
|
+
annotations: Base::DESTRUCTIVE,
|
|
73
77
|
inputSchema: {
|
|
74
78
|
type: "object",
|
|
75
79
|
properties: {
|
|
@@ -84,12 +88,14 @@ module CompletionKit
|
|
|
84
88
|
handler: :update
|
|
85
89
|
},
|
|
86
90
|
"metrics_delete" => {
|
|
87
|
-
description: "
|
|
91
|
+
description: "Permanently delete a metric. This also deletes all of its versions and drafts and every human agree, disagree or borderline verdict recorded against them. It removes the metric from every run and metric group and unlinks its tags, though the tags themselves stay. Scores it already gave stay on their responses but are no longer linked to the metric. This cannot be undone. Returns a confirmation message.",
|
|
92
|
+
annotations: Base::DESTRUCTIVE,
|
|
88
93
|
inputSchema: {type: "object", properties: {id: {type: "integer"}}, required: ["id"]},
|
|
89
94
|
handler: :delete
|
|
90
95
|
},
|
|
91
96
|
"metrics_suggest_variants" => {
|
|
92
|
-
description: "
|
|
97
|
+
description: "Get AI suggestions to improve an LLM judge that disagrees with people. It sends the judge's current instruction and rubric, plus up to 8 recent responses where a person disagreed with the judge's score and up to 6 that a person marked borderline (each input and output cut to 200 characters), to your own model provider. The model is asked for rewrites of the instruction, the rubric or both that close the gap. Each rewrite is saved as a draft metric version with source \"suggestion\", and earlier suggestion drafts are kept. The live judge does not change until you publish a draft with metric_versions_publish. Nothing tests a draft before you publish it. Once it is live and people have recorded verdicts on it with agreements_create, judges_compare shows whether it agrees with people more often than the version it replaced. Returns the saved drafts. It works best after you have recorded disagreements with agreements_create. Checks are refused, because they are exact and have nothing to suggest. count defaults to 1 and is capped at 3. model picks the model that writes the rewrites and defaults to your judge model. The call uses your provider key and runs while you wait. It fails with an error if no judging model is set up or the model you name is unknown. If the model's reply can't be read, nothing is saved and you get a \"no parseable variants\" error. A call your provider refuses, such as one with a bad key, usually ends in that same error.",
|
|
98
|
+
annotations: Base::ADDITIVE,
|
|
93
99
|
inputSchema: {
|
|
94
100
|
type: "object",
|
|
95
101
|
properties: {
|
|
@@ -5,17 +5,20 @@ module CompletionKit
|
|
|
5
5
|
|
|
6
6
|
TOOLS = {
|
|
7
7
|
"prompts_list" => {
|
|
8
|
-
description: "List
|
|
8
|
+
description: "List every prompt in CompletionKit, newest first. Each version of a prompt is its own entry, so drafts and older versions appear next to the published one. Versions of the same prompt share a family_key, and current=true marks the published version that apps get when they fetch the prompt by name or family_key. Returns each version's id, name, description, template, llm_model, family_key, version_number, current, timestamps and tags. Use it to find a prompt's ID before you test it, edit it, compare its versions or ask for a suggested improvement.",
|
|
9
|
+
annotations: Base::READ_ONLY,
|
|
9
10
|
inputSchema: {type: "object", properties: {}, required: []},
|
|
10
11
|
handler: :list
|
|
11
12
|
},
|
|
12
13
|
"prompts_get" => {
|
|
13
|
-
description: "Get
|
|
14
|
+
description: "Get one prompt version by its numeric ID. Returns its name, description, template text with {{variable}} placeholders, the model that answers it (llm_model), family_key, version_number, whether it is the published version (current), timestamps and tags. Use it to read a prompt before you edit it, test it in a run or compare it with another version.",
|
|
15
|
+
annotations: Base::READ_ONLY,
|
|
14
16
|
inputSchema: {type: "object", properties: {id: {type: "integer", description: "Prompt ID"}}, required: ["id"]},
|
|
15
17
|
handler: :get
|
|
16
18
|
},
|
|
17
19
|
"prompts_create" => {
|
|
18
|
-
description: "Create a prompt",
|
|
20
|
+
description: "Create a new prompt to test. Give it a name, a template with {{variable}} placeholders that match your dataset's column names, and the model that should answer it (llm_model). The model name is checked against the models your saved provider keys serve. If CompletionKit has not found any models from a saved key yet, any model name is accepted, and runs fail until a key for that model's provider is added. The new prompt is version 1 of a new prompt family and is published right away. If another published prompt already has the same name, apps that fetch prompts by name get this new one instead. To change an existing prompt, use prompts_update so its versions stay together. tag_names creates any tags that don't exist yet. Returns the saved prompt with its ID. To test it, create a run with runs_create and start it with runs_generate.",
|
|
21
|
+
annotations: Base::DESTRUCTIVE,
|
|
19
22
|
inputSchema: {
|
|
20
23
|
type: "object",
|
|
21
24
|
properties: {
|
|
@@ -28,7 +31,8 @@ module CompletionKit
|
|
|
28
31
|
handler: :create
|
|
29
32
|
},
|
|
30
33
|
"prompts_update" => {
|
|
31
|
-
description: "
|
|
34
|
+
description: "Edit a prompt's name, description, template, model or tags. Changing the template or model of the published version, or editing any version that already has runs, creates a new DRAFT version (current=false) instead of changing it, so an agent's edits don't go live without a gate. The draft copies the old version's name, description, template and model, with your changes applied. Run and compare the draft, then promote it with prompts_publish. An unpublished draft with no runs, or a name, description or tag change on a published version with no runs, is updated in place and the old values are not kept. Renaming a published version in place takes effect at once, so apps that fetch it by its old name stop finding it, while fetches by family_key or ID still work. tag_names replaces the whole tag list. A new draft does not copy the old version's tags, so pass tag_names to keep them, or the draft has no tags, even after it is published. Returns the saved version, which has a new ID when a draft was created.",
|
|
35
|
+
annotations: Base::DESTRUCTIVE,
|
|
32
36
|
inputSchema: {
|
|
33
37
|
type: "object",
|
|
34
38
|
properties: {
|
|
@@ -41,17 +45,20 @@ module CompletionKit
|
|
|
41
45
|
handler: :update
|
|
42
46
|
},
|
|
43
47
|
"prompts_delete" => {
|
|
44
|
-
description: "
|
|
48
|
+
description: "Permanently delete one prompt version by ID, along with every run of that version and all of those runs' responses, scores, judge reviews, agreements and suggestions. Other versions of the prompt, and your datasets, metrics and tags, are kept. Apps that fetch this version by its ID get a not-found error. If you delete the published version, no other version is published in its place, so apps that fetch the prompt by family_key get a not-found error, and apps that fetch it by name get a not-found error or another published prompt with the same name, until you publish another version with prompts_publish. This cannot be undone. Returns a confirmation message.",
|
|
49
|
+
annotations: Base::DESTRUCTIVE,
|
|
45
50
|
inputSchema: {type: "object", properties: {id: {type: "integer"}}, required: ["id"]},
|
|
46
51
|
handler: :delete
|
|
47
52
|
},
|
|
48
53
|
"prompts_publish" => {
|
|
49
|
-
description: "Publish a prompt version, making it the current version",
|
|
54
|
+
description: "Publish a prompt version, making it the current version. Apps that fetch the prompt by its name or family_key get this version from their next request, and the version that was published before stops being current. Use it to ship a draft from prompts_update once a run shows it scores at least as well, or to roll back by publishing an older version. Returns the published version.",
|
|
55
|
+
annotations: Base::DESTRUCTIVE,
|
|
50
56
|
inputSchema: {type: "object", properties: {id: {type: "integer"}}, required: ["id"]},
|
|
51
57
|
handler: :publish
|
|
52
58
|
},
|
|
53
59
|
"prompts_suggest_improvement" => {
|
|
54
|
-
description: "Suggest an improved version of a prompt, grounded in a run's test results and judge feedback.
|
|
60
|
+
description: "Suggest an improved version of a prompt, grounded in a run's test results and judge feedback. It reads up to 20 of the run's responses with their scores and reviews, then asks the prompt's own model, on your saved provider key, for reasoning plus a rewritten template that keeps the {{variables}}. It then tests the rewrite on up to 30 of the run's judge-scored rows and scores the new answers with the run's LLM-judge metrics and judge model. Returns suggestion_id, prompt_id, reasoning, suggested_template, original_template, validation (before and after averages and how many rows improved, regressed or stayed the same) and net_negative, which is true when the rewrite's average score fell or more rows got worse than better. If validation.tested is 0, nothing was checked and net_negative is false. That happens when the run has no LLM-judge metrics, has no judge-scored rows with both input and output, or every test call failed. It saves the result as a Suggestion. The prompt itself is not changed, so pass suggested_template to prompts_update to save it as a draft version. Requires a run that has a prompt (not a scoring-only run). Each call can make dozens of model calls on your saved provider keys and can take a while.",
|
|
61
|
+
annotations: Base::ADDITIVE,
|
|
55
62
|
inputSchema: {
|
|
56
63
|
type: "object",
|
|
57
64
|
properties: {run_id: {type: "integer", description: "The run whose results ground the improvement."}},
|
|
@@ -5,17 +5,20 @@ module CompletionKit
|
|
|
5
5
|
|
|
6
6
|
TOOLS = {
|
|
7
7
|
"provider_credentials_list" => {
|
|
8
|
-
description: "List
|
|
8
|
+
description: "List the model provider connections saved in your CompletionKit account (OpenAI, Anthropic, Ollama or another OpenAI-compatible endpoint, OpenRouter, Azure AI Foundry). Returns each one's id, provider, custom endpoint, API version and timestamps. API keys are not exposed. Use it to see which providers are connected before you test a prompt, start a run or use an LLM judge. It does not show whether a key is still stored or whether the provider's models loaded.",
|
|
9
|
+
annotations: Base::READ_ONLY,
|
|
9
10
|
inputSchema: {type: "object", properties: {}, required: []},
|
|
10
11
|
handler: :list
|
|
11
12
|
},
|
|
12
13
|
"provider_credentials_get" => {
|
|
13
|
-
description: "Get
|
|
14
|
+
description: "Get one saved model provider connection by its ID. Returns its provider, custom endpoint, API version and timestamps. The API key is not exposed. Use it to check how a provider is set up before you change it.",
|
|
15
|
+
annotations: Base::READ_ONLY,
|
|
14
16
|
inputSchema: {type: "object", properties: {id: {type: "integer"}}, required: ["id"]},
|
|
15
17
|
handler: :get
|
|
16
18
|
},
|
|
17
19
|
"provider_credentials_create" => {
|
|
18
|
-
description: "
|
|
20
|
+
description: "Save an API key for a model provider (openai, anthropic, ollama for Ollama or any other OpenAI-compatible endpoint, openrouter or azure_foundry) so CompletionKit can run your prompts and LLM judges with it. You can save one credential per provider. azure_foundry requires api_endpoint, ollama needs api_endpoint for its models to load, and any api_endpoint must be an http(s) URL that resolves to a public address. The key is stored encrypted and is never returned. Saving starts a background job that lists the provider's models. For every provider except openrouter, the job then sends each model that has not been tested yet a short test prompt to check it can generate text and, if it can, a second one to check it can judge, and these requests use your key. Returns the saved credential without the key.",
|
|
21
|
+
annotations: Base::OPEN_WORLD_ADDITIVE,
|
|
19
22
|
inputSchema: {
|
|
20
23
|
type: "object",
|
|
21
24
|
properties: {
|
|
@@ -29,7 +32,8 @@ module CompletionKit
|
|
|
29
32
|
handler: :create
|
|
30
33
|
},
|
|
31
34
|
"provider_credentials_update" => {
|
|
32
|
-
description: "
|
|
35
|
+
description: "Change a saved model provider connection: its API key, provider, custom endpoint or API version. A new api_key replaces the stored key, and the old key cannot be recovered. An empty api_key clears the stored key. Changing provider or api_endpoint on a credential with a stored key requires api_key in the same call, so a stored key never moves to a new provider or endpoint. Each provider can have only one credential. Any api_endpoint must be an http(s) URL that resolves to a public address. Each successful save runs model discovery again in the background. It lists the provider's models and, for every provider except openrouter, sends each model that has not been tested yet a short test prompt to check it can generate text and, if it can, a second one to check it can judge, using the key. Models the provider no longer lists are marked retired, and prompts or judges set to a retired model can fail to run. Returns the credential without the key.",
|
|
36
|
+
annotations: Base::OPEN_WORLD_DESTRUCTIVE,
|
|
33
37
|
inputSchema: {
|
|
34
38
|
type: "object",
|
|
35
39
|
properties: {
|
|
@@ -42,7 +46,8 @@ module CompletionKit
|
|
|
42
46
|
handler: :update
|
|
43
47
|
},
|
|
44
48
|
"provider_credentials_delete" => {
|
|
45
|
-
description: "
|
|
49
|
+
description: "Permanently delete a saved model provider connection and its stored API key, and remove that provider's models from your model list. It is refused with an error while any current prompt version uses one of that provider's models, or while any run, including past runs, used one of them as its judge. This cannot be undone, so confirm with the user before calling it. Returns a short confirmation.",
|
|
50
|
+
annotations: Base::DESTRUCTIVE,
|
|
46
51
|
inputSchema: {type: "object", properties: {id: {type: "integer"}}, required: ["id"]},
|
|
47
52
|
handler: :delete
|
|
48
53
|
}
|
|
@@ -10,11 +10,8 @@ module CompletionKit
|
|
|
10
10
|
|
|
11
11
|
TOOLS = {
|
|
12
12
|
"responses_list" => {
|
|
13
|
-
description: "List responses
|
|
14
|
-
|
|
15
|
-
"because full payloads are large: use \"fields\" to drop the bodies, \"min_score\"/\"max_score\" " \
|
|
16
|
-
"to isolate low scorers, and sort \"score_asc\" to read the worst rows first. For per-metric " \
|
|
17
|
-
"averages of the whole run use runs_get instead of aggregating here.",
|
|
13
|
+
description: "List the AI responses in a run with their scores, in row order, to see which test cases scored well or badly and why. Each row has the input, the response_text, the expected_output, an average judge score from 1 to 5 (5 is best), and each metric's review with judge feedback or a check's pass/fail. Returns {total, limit, offset, returned, responses}. Defaults to #{Base::DEFAULT_PAGE_LIMIT} rows because full payloads are large: use \"fields\" to drop the bodies, \"min_score\"/\"max_score\" to isolate low scorers, and sort \"score_asc\" to read the worst rows first. Rows with no judge score, such as rows graded only by checks, are left out by a score filter and listed last by a score sort. For per-metric averages of the whole run use runs_get instead of aggregating here.",
|
|
14
|
+
annotations: Base::READ_ONLY,
|
|
18
15
|
inputSchema: {
|
|
19
16
|
type: "object",
|
|
20
17
|
properties: {
|
|
@@ -32,7 +29,8 @@ module CompletionKit
|
|
|
32
29
|
handler: :list
|
|
33
30
|
},
|
|
34
31
|
"responses_get" => {
|
|
35
|
-
description: "Get a
|
|
32
|
+
description: "Get one response from a run in full, by run_id and response id. The response must belong to that run. Returns the input, the complete response_text, the expected_output, the average judge score from 1 to 5 (5 is best), each metric's review with the judge's score and feedback or the check's pass/fail, the row's status and attempts, and any provider error. Use it to read a single test case closely after responses_list points you to it.",
|
|
33
|
+
annotations: Base::READ_ONLY,
|
|
36
34
|
inputSchema: {
|
|
37
35
|
type: "object",
|
|
38
36
|
properties: {run_id: {type: "integer"}, id: {type: "integer"}},
|
|
@@ -22,19 +22,20 @@ module CompletionKit
|
|
|
22
22
|
|
|
23
23
|
TOOLS = {
|
|
24
24
|
"runs_list" => {
|
|
25
|
-
description: "List
|
|
25
|
+
description: "List your test runs, newest first. A run tests one prompt version against a dataset, or grades existing outputs, and scores each response with your metrics. It returns each run's id, name, status (pending, running, completed or failed), prompt and dataset ids, judge model, average score (1 to 5, 5 is best), check pass rate, per-metric averages and progress. Runs older than the account's history window may be left out. Use it to find a run's id or see recent results, then call runs_get for one run.",
|
|
26
|
+
annotations: Base::READ_ONLY,
|
|
26
27
|
inputSchema: {type: "object", properties: {}, required: []},
|
|
27
28
|
handler: :list
|
|
28
29
|
},
|
|
29
30
|
"runs_get" => {
|
|
30
|
-
description: "Get
|
|
31
|
-
|
|
32
|
-
"Use this to find the metric dragging a prompt down without listing responses.",
|
|
31
|
+
description: "Get one test run's results by id: status, progress, average score (1 to 5, 5 is best), check pass rate, the ids of failed responses, any failure reason, and \"metric_averages\", a per-metric breakdown with each metric's average score (or pass rate for checks), how many rows it graded, and how many scored low. Use this to see how a prompt scored and to find the metric dragging a prompt down without listing responses. Poll it after runs_generate, runs_regrade or runs_rerun until status is completed or failed. The result includes a warning when no metrics are attached or when the judge's scores are not reproducible.",
|
|
32
|
+
annotations: Base::READ_ONLY,
|
|
33
33
|
inputSchema: {type: "object", properties: {id: {type: "integer"}}, required: ["id"]},
|
|
34
34
|
handler: :get
|
|
35
35
|
},
|
|
36
36
|
"runs_create" => {
|
|
37
|
-
description: "
|
|
37
|
+
description: "Set up a test run that scores a prompt's responses on a dataset. Give prompt_id and dataset_id to test a prompt: each dataset row fills the prompt's variables, the prompt's model answers it, and that answer gets scored. Omit prompt_id and provide output_column to score existing outputs by grading a pre-existing dataset column instead of generating new ones. Attach metrics with metric_ids or metric_group_id, and set judge_model, or LLM-judged metrics are skipped; with no metrics the run scores nothing. It fails if the dataset is missing a column the run needs, such as one the prompt uses, or if no configured provider serves judge_model. This only creates the run; call runs_generate to start it. It returns the new run, with a warning if no metrics are attached or if judge_temperature is above 0, which makes scores irreproducible.",
|
|
38
|
+
annotations: Base::ADDITIVE,
|
|
38
39
|
inputSchema: {
|
|
39
40
|
type: "object",
|
|
40
41
|
properties: {
|
|
@@ -54,7 +55,8 @@ module CompletionKit
|
|
|
54
55
|
handler: :create
|
|
55
56
|
},
|
|
56
57
|
"runs_update" => {
|
|
57
|
-
description: "
|
|
58
|
+
description: "Change a test run's settings: name, dataset, judge model, temperature, judge temperature, max tokens, output and expected columns, metrics or tags. The prompt cannot be changed. Passing metric_ids or metric_group_id replaces the run's whole metric list, and tag_names replaces all of its tags. It does not regenerate or rescore anything, and existing responses and scores stay as they are; call runs_regrade to rescore with the new metrics, or runs_rerun for a fresh run. It returns the updated run with the same warnings as runs_get.",
|
|
59
|
+
annotations: Base::DESTRUCTIVE,
|
|
58
60
|
inputSchema: {
|
|
59
61
|
type: "object",
|
|
60
62
|
properties: {
|
|
@@ -74,27 +76,32 @@ module CompletionKit
|
|
|
74
76
|
handler: :update
|
|
75
77
|
},
|
|
76
78
|
"runs_delete" => {
|
|
77
|
-
description: "
|
|
79
|
+
description: "Permanently delete a test run and everything in it: its generated responses, their scores and judge feedback, the human agreement verdicts recorded on it (which also drops them from the judge's agreement stats), and any prompt suggestions made from it. This cannot be undone. The prompt, dataset and metrics are kept. Confirm with the user before calling. It returns a short confirmation message.",
|
|
80
|
+
annotations: Base::DESTRUCTIVE,
|
|
78
81
|
inputSchema: {type: "object", properties: {id: {type: "integer"}}, required: ["id"]},
|
|
79
82
|
handler: :delete
|
|
80
83
|
},
|
|
81
84
|
"runs_generate" => {
|
|
82
|
-
description: "Start a run.
|
|
85
|
+
description: "Start a test run so it produces and scores responses. It is required for every run, including score-only runs (no prompt): it generates responses with the prompt when there is one, and otherwise copies the graded dataset column and grades it. In a prompt run each dataset row goes to the prompt's model, and in either kind of run each LLM-judged metric then scores the result with the run's judge model. LLM-judged metrics are skipped when the run has no judge model. These model calls go to the providers you configured, which may charge for them; checks run without a provider call. The work runs in the background, so poll runs_get until status is completed or failed, and do not call this tool again to check progress. Only a pending or failed run can start, and any refused start marks the run as failed. Calling it on a running or completed run therefore does not start it but marks that run as failed, and a second call would then delete its results. Restarting a failed run first deletes its earlier responses, their scores and any human verdicts on them. For a completed run use runs_rerun, runs_retry_failures or runs_regrade instead. It returns the run with its progress, or the reason it could not start, such as a missing provider key or an empty dataset.",
|
|
86
|
+
annotations: Base::DESTRUCTIVE,
|
|
83
87
|
inputSchema: {type: "object", properties: {id: {type: "integer"}}, required: ["id"]},
|
|
84
88
|
handler: :generate
|
|
85
89
|
},
|
|
86
90
|
"runs_regrade" => {
|
|
87
|
-
description: "Re-
|
|
91
|
+
description: "Re-score a test run's existing responses with its currently attached metrics, without regenerating. Use it after attaching or editing metrics on an already-generated run. It first clears every existing score and judge comment on the run's succeeded responses, including scores for metrics no longer attached, so the earlier scores are lost. If the run has checks but no judge model, its LLM-judged scores are cleared and not replaced. Human verdicts already recorded on the run are kept even though people gave them on the earlier scores, and agreement stats then pair them with the new scores. LLM-judged metrics call your judge model's provider; checks do not. The work runs in the background, so poll runs_get for progress. It returns the run, or an error when no configured provider serves the judge model, or when there is nothing to re-grade because the run has no succeeded responses, no metrics, or only LLM-judged metrics and no working judge model.",
|
|
92
|
+
annotations: Base::DESTRUCTIVE,
|
|
88
93
|
inputSchema: {type: "object", properties: {id: {type: "integer"}}, required: ["id"]},
|
|
89
94
|
handler: :regrade
|
|
90
95
|
},
|
|
91
96
|
"runs_rerun" => {
|
|
92
|
-
description: "
|
|
97
|
+
description: "Copy a test run and start the copy, with the same prompt version, dataset, metrics, judge model, settings and tags, under a new automatic name. Use it when the judge changed and you want a clean run instead of mixing versions, or to repeat a test. The original run and its scores stay untouched. The copy runs in the background and calls your model providers the same way runs_generate does. To test a different prompt version, create a new run with runs_create instead. It returns the new run; poll runs_get with its id for progress. If the prompt's model or the judge model has no configured provider, or the dataset no longer has a column the run needs, it returns an error and creates nothing. If the copy is created but cannot start, for example because the dataset is empty, it returns the reason and the copy stays as a failed run.",
|
|
98
|
+
annotations: Base::ADDITIVE,
|
|
93
99
|
inputSchema: {type: "object", properties: {id: {type: "integer"}}, required: ["id"]},
|
|
94
100
|
handler: :rerun
|
|
95
101
|
},
|
|
96
102
|
"runs_retry_failures" => {
|
|
97
|
-
description: "
|
|
103
|
+
description: "Retry only the parts of a test run that failed, such as provider timeouts, rate limits or judge errors. Failed responses are generated again and failed scores are retried, while responses and scores that succeeded are left alone; optionally limit it to specific response ids via \"only\". It is refused when the judge has changed since the run executed, meaning a metric on the run now has a newer version than the one that scored it, because a retry would mix versions in the same run; use runs_rerun instead. The retries run in the background and call your model providers, except for checks. It returns the run with its progress, or the run unchanged when nothing failed.",
|
|
104
|
+
annotations: Base::ADDITIVE,
|
|
98
105
|
inputSchema: {type: "object", properties: {id: {type: "integer"}, only: {type: "array", items: {type: "integer"}}}, required: ["id"]},
|
|
99
106
|
handler: :retry_failures
|
|
100
107
|
}
|
|
@@ -5,17 +5,20 @@ module CompletionKit
|
|
|
5
5
|
|
|
6
6
|
TOOLS = {
|
|
7
7
|
"tags_list" => {
|
|
8
|
-
description: "List
|
|
8
|
+
description: "List every tag in CompletionKit, in name order. Returns each tag's id, name, color and timestamps. Tags label prompts, datasets, metrics, metric groups and runs. Use this to see which labels exist, or to find a tag's ID before renaming or deleting it.",
|
|
9
|
+
annotations: Base::READ_ONLY,
|
|
9
10
|
inputSchema: {type: "object", properties: {}, required: []},
|
|
10
11
|
handler: :list
|
|
11
12
|
},
|
|
12
13
|
"tags_get" => {
|
|
13
|
-
description: "Get
|
|
14
|
+
description: "Get one tag by ID. Returns its id, name, color and timestamps. Use it to check a tag's current name before you rename it or pass it in tag_names.",
|
|
15
|
+
annotations: Base::READ_ONLY,
|
|
14
16
|
inputSchema: {type: "object", properties: {id: {type: "integer"}}, required: ["id"]},
|
|
15
17
|
handler: :get
|
|
16
18
|
},
|
|
17
19
|
"tags_create" => {
|
|
18
|
-
description: "Create a tag. Color is auto-assigned.",
|
|
20
|
+
description: "Create a tag for labelling prompts, datasets, metrics, metric groups and runs. The name is trimmed and lowercased. It may contain only the letters a to z, digits, spaces, hyphens and underscores (accented and non-Latin letters are rejected), can be up to 64 characters, and must not match an existing tag. Color is auto-assigned. Returns the new tag. You don't need this tool before tagging a record, because the create and update tools for those records take tag_names and create missing tags for you.",
|
|
21
|
+
annotations: Base::ADDITIVE,
|
|
19
22
|
inputSchema: {
|
|
20
23
|
type: "object",
|
|
21
24
|
properties: {name: {type: "string"}},
|
|
@@ -24,7 +27,8 @@ module CompletionKit
|
|
|
24
27
|
handler: :create
|
|
25
28
|
},
|
|
26
29
|
"tags_update" => {
|
|
27
|
-
description: "Rename a tag.",
|
|
30
|
+
description: "Rename a tag. Pass the new name; leaving it out returns an error. The name follows the same rules as tags_create and can't match another tag, because tags are not merged. The new name shows at once on every prompt, dataset, metric, metric group and run that carries the tag. Returns the updated tag.",
|
|
31
|
+
annotations: Base::DESTRUCTIVE,
|
|
28
32
|
inputSchema: {
|
|
29
33
|
type: "object",
|
|
30
34
|
properties: {id: {type: "integer"}, name: {type: "string"}},
|
|
@@ -33,7 +37,8 @@ module CompletionKit
|
|
|
33
37
|
handler: :update
|
|
34
38
|
},
|
|
35
39
|
"tags_delete" => {
|
|
36
|
-
description: "Delete a tag. Removes the tag from every linked metric, prompt, run, and dataset.",
|
|
40
|
+
description: "Delete a tag. Removes the tag from every linked metric, metric group, prompt, run, and dataset; those records themselves are kept. This cannot be undone. Use tags_update instead if you only want to rename it. Returns a confirmation message.",
|
|
41
|
+
annotations: Base::DESTRUCTIVE,
|
|
37
42
|
inputSchema: {type: "object", properties: {id: {type: "integer"}}, required: ["id"]},
|
|
38
43
|
handler: :delete
|
|
39
44
|
}
|
|
@@ -372,11 +372,9 @@
|
|
|
372
372
|
<div class="ck-api-endpoint" style="padding-top: 1rem;">
|
|
373
373
|
<p class="ck-kicker" style="margin-bottom: 0.5rem;">MCP tools</p>
|
|
374
374
|
<div class="ck-mcp-tools">
|
|
375
|
-
|
|
376
|
-
|
|
377
|
-
|
|
378
|
-
<div class="ck-mcp-tool"><code class="ck-mcp-tool__name">tags_update</code><span class="ck-mcp-tool__desc">Update a tag's name</span></div>
|
|
379
|
-
<div class="ck-mcp-tool"><code class="ck-mcp-tool__name">tags_delete</code><span class="ck-mcp-tool__desc">Delete a tag and remove all its taggings</span></div>
|
|
375
|
+
<% CompletionKit::McpTools::Tags.definitions.each do |tool| %>
|
|
376
|
+
<div class="ck-mcp-tool"><code class="ck-mcp-tool__name"><%= tool[:name] %></code><span class="ck-mcp-tool__desc"><%= tool[:description] %></span></div>
|
|
377
|
+
<% end %>
|
|
380
378
|
</div>
|
|
381
379
|
<p class="ck-meta-copy" style="margin-top: 0.75rem;">The existing <code>metrics_create</code>, <code>metrics_update</code>, <code>prompts_create</code>, <code>prompts_update</code>, <code>runs_create</code>, <code>runs_update</code>, <code>datasets_create</code>, <code>datasets_create_from_url</code>, <code>datasets_update</code>, <code>metric_groups_create</code>, and <code>metric_groups_update</code> tools all accept a <code>tag_names</code> parameter with the same auto-create and replace semantics as the REST API.</p>
|
|
382
380
|
</div>
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
<% pc_delete_id = "ck_delete_provider_#{provider_credential.id}" %>
|
|
3
3
|
<% pc_delete_confirm = "Delete the #{provider_credential.display_provider} provider and its discovered models? This can't be undone." %>
|
|
4
4
|
<% end %>
|
|
5
|
-
<%= form_with(model: provider_credential, local: true) do |form| %>
|
|
5
|
+
<%= form_with(model: provider_credential, url: local_assigns[:url], local: true) do |form| %>
|
|
6
6
|
<% if provider_credential.errors.any? %>
|
|
7
7
|
<div class="ck-flash ck-flash--alert" role="alert">
|
|
8
8
|
<p class="ck-flash__title"><%= pluralize(provider_credential.errors.count, "problem") %> prevented this provider credential from being saved.</p>
|
|
@@ -66,7 +66,9 @@
|
|
|
66
66
|
<% if provider_credential.persisted? %>
|
|
67
67
|
<%= ck_delete_trigger(form_id: pc_delete_id, label: "Delete provider", confirm: pc_delete_confirm) %>
|
|
68
68
|
<% end %>
|
|
69
|
-
|
|
69
|
+
<% unless local_assigns[:cancel_path] == false %>
|
|
70
|
+
<%= link_to "Cancel", local_assigns[:cancel_path] || provider_credentials_path, class: ck_button_classes(:light, variant: :outline), tabindex: "0" %>
|
|
71
|
+
<% end %>
|
|
70
72
|
<%= form.submit(provider_credential.persisted? ? "Save provider" : "Create provider", class: ck_button_classes(:dark)) %>
|
|
71
73
|
</div>
|
|
72
74
|
|
data/lib/completion_kit.rb
CHANGED
|
@@ -13,7 +13,7 @@ module CompletionKit
|
|
|
13
13
|
attr_accessor :tenant_scope, :tenant_scope_columns
|
|
14
14
|
attr_accessor :api_reference_authentication_partial
|
|
15
15
|
attr_accessor :runs_display_scope, :runs_display_footer_partial
|
|
16
|
-
attr_accessor :api_rate_limit, :web_rate_limit, :max_upload_bytes
|
|
16
|
+
attr_accessor :api_rate_limit, :api_rate_limit_key, :web_rate_limit, :max_upload_bytes
|
|
17
17
|
attr_accessor :max_template_bytes
|
|
18
18
|
attr_reader :regex_timeout
|
|
19
19
|
attr_accessor :allow_loopback_endpoints
|
|
@@ -33,6 +33,7 @@ module CompletionKit
|
|
|
33
33
|
@medium_quality_threshold = 3
|
|
34
34
|
|
|
35
35
|
@api_rate_limit = 120
|
|
36
|
+
@api_rate_limit_key = ->(request) { request.remote_ip }
|
|
36
37
|
@web_rate_limit = 300
|
|
37
38
|
@max_upload_bytes = 25 * 1024 * 1024
|
|
38
39
|
@max_template_bytes = 1024 * 1024
|